[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:20 dist-test-slave-3mqx krb5kdc[22150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:20 dist-test-slave-3mqx krb5kdc[22150](info): set up 2 sockets
Jun 20 14:53:20 dist-test-slave-3mqx krb5kdc[22150](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:53:21.296941 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:21 dist-test-slave-3mqx krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:21.484193 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38089
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:38089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:21.822261 22163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:21.822650 22163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:21.822821 22163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:21.830559 22163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:21.830736 22163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:21.830857 22163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:21.830967 22163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:21.831055 22163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:21.841636 22163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:38089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:21.843278 22163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:21.844718 22163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:21.858783 22171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:21.858896 22169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:21.858953 22168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:21.859227 22163 server_base.cc:1034] running on GCE node
I20260620 14:53:21.860514 22163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:21.862123 22163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:21.863384 22163 hybrid_clock.cc:648] HybridClock initialized: now 1781967201863333 us; error 45 us; skew 500 ppm
I20260620 14:53:21.863829 22163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:21 dist-test-slave-3mqx krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967201, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:21.867465 22163 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:21.869237 22163 webserver.cc:460] Webserver started at http://127.21.159.254:44077/ using document root <none> and password file <none>
I20260620 14:53:21.870042 22163 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:21.870215 22163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:21.870568 22163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:21.874387 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "6905877b4f684fdb8964955778b88f1a"
format_stamp: "Formatted at 2026-06-20 14:53:21 on dist-test-slave-3mqx"
server_key: "168579ff486d2c003a970a883ababe0d"
server_key_iv: "66a06955a27545ac6545b764801faa41"
server_key_version: "encryptionkey@0"
I20260620 14:53:21.875489 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "6905877b4f684fdb8964955778b88f1a"
format_stamp: "Formatted at 2026-06-20 14:53:21 on dist-test-slave-3mqx"
server_key: "168579ff486d2c003a970a883ababe0d"
server_key_iv: "66a06955a27545ac6545b764801faa41"
server_key_version: "encryptionkey@0"
I20260620 14:53:21.881942 22163 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:53:21.886301 22178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:21.888103 22163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 14:53:21.888422 22163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "6905877b4f684fdb8964955778b88f1a"
format_stamp: "Formatted at 2026-06-20 14:53:21 on dist-test-slave-3mqx"
server_key: "168579ff486d2c003a970a883ababe0d"
server_key_iv: "66a06955a27545ac6545b764801faa41"
server_key_version: "encryptionkey@0"
I20260620 14:53:21.888671 22163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:21.912357 22163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:21.917249 22163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:21.917671 22163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:21.938086 22163 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:38089
I20260620 14:53:21.938125 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:38089 every 8 connection(s)
I20260620 14:53:21.939687 22163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:21.943779 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 22163
I20260620 14:53:21.944118 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:21.944434 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:21.946461 22143 external_mini_cluster.cc:1428] Setting key 3caf53d56247062a10bd20a210909427
I20260620 14:53:21.956786 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a: Bootstrap starting.
I20260620 14:53:21.959913 22230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:21.961076 22230 log.cc:824] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:21.964056 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a: No bootstrap required, opened a new log
I20260620 14:53:21.971391 22230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905877b4f684fdb8964955778b88f1a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38089 } }
I20260620 14:53:21.971815 22230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:21.971935 22230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6905877b4f684fdb8964955778b88f1a, State: Initialized, Role: FOLLOWER
I20260620 14:53:21.972607 22230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905877b4f684fdb8964955778b88f1a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38089 } }
I20260620 14:53:21.972862 22230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:21.973008 22230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:21.973201 22230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:53:21 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:21.976169 22230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905877b4f684fdb8964955778b88f1a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38089 } }
I20260620 14:53:21.976840 22230 leader_election.cc:302] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6905877b4f684fdb8964955778b88f1a; no voters:
I20260620 14:53:21.977563 22230 leader_election.cc:288] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:21.977732 22235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:21.978454 22235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [term 1 LEADER]: Becoming Leader. State: Replica: 6905877b4f684fdb8964955778b88f1a, State: Running, Role: LEADER
I20260620 14:53:21.979156 22235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905877b4f684fdb8964955778b88f1a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38089 } }
I20260620 14:53:21.980751 22230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:21.981361 22235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6905877b4f684fdb8964955778b88f1a. Latest consensus state: current_term: 1 leader_uuid: "6905877b4f684fdb8964955778b88f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905877b4f684fdb8964955778b88f1a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38089 } } }
I20260620 14:53:21.981611 22235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:21.982157 22237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6905877b4f684fdb8964955778b88f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6905877b4f684fdb8964955778b88f1a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38089 } } }
I20260620 14:53:21.982549 22237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:21.984651 22239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:21.991765 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:21.950464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51848 (local address 127.21.159.254:38089)
0620 14:53:21.951350 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:21.951369 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:21.952511 (+ 1142us) server_negotiation.cc:407] Connection header received
0620 14:53:21.962098 (+ 9587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:21.962131 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:21.962612 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:21.963026 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:21.965940 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:21.967396 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:21.969086 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:21.969807 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:21.970056 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:21.976929 (+ 6873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:21.976939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:21.976955 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:21.977041 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:21.981531 (+ 4490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:21.983212 (+ 1681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:21.983225 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:21.983237 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:21.983526 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:21.984242 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:21.984254 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:21.984260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:21.985237 (+ 977us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:21.985640 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:21.989407 (+ 3767us) server_negotiation.cc:299] Negotiation successful
0620 14:53:21.989964 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":392,"threads_started":1}
I20260620 14:53:21.997356 22239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:22.009518 22239 catalog_manager.cc:1347] Generated new cluster ID: 3c66a8454c8d41cd8f4a36d7ab2d8581
I20260620 14:53:22.009644 22239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:22.021248 22239 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:22.022759 22239 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:22.032437 22239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6905877b4f684fdb8964955778b88f1a: Generated new TSK 0
I20260620 14:53:22.033576 22239 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:22.084604 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38089
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:22.450209 22256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:22.450584 22256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:22.450759 22256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:22.458595 22256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:22.458779 22256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:22.458971 22256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:22.469200 22256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38089
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:22.470654 22256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:22.472184 22256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:22.487111 22261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:22.487145 22262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:22.487185 22264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:22.488080 22256 server_base.cc:1034] running on GCE node
I20260620 14:53:22.488960 22256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:22.490119 22256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:22.491416 22256 hybrid_clock.cc:648] HybridClock initialized: now 1781967202491367 us; error 69 us; skew 500 ppm
I20260620 14:53:22.492296 22256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:22 dist-test-slave-3mqx krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967202, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:22.496202 22256 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:22.497996 22256 webserver.cc:460] Webserver started at http://127.21.159.193:42909/ using document root <none> and password file <none>
I20260620 14:53:22.498839 22256 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:22.499027 22256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:22.499439 22256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:22.503003 22256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "c5592c34e2e64e61bf4d139ae4be4a7c"
format_stamp: "Formatted at 2026-06-20 14:53:22 on dist-test-slave-3mqx"
server_key: "409a92177e13f583a7ff6a0efafc369b"
server_key_iv: "f8dcd32b0d4647c699f5cb6af8d44b0d"
server_key_version: "encryptionkey@0"
I20260620 14:53:22.504086 22256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "c5592c34e2e64e61bf4d139ae4be4a7c"
format_stamp: "Formatted at 2026-06-20 14:53:22 on dist-test-slave-3mqx"
server_key: "409a92177e13f583a7ff6a0efafc369b"
server_key_iv: "f8dcd32b0d4647c699f5cb6af8d44b0d"
server_key_version: "encryptionkey@0"
I20260620 14:53:22.510110 22256 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:53:22.514550 22271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:22.516376 22256 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 14:53:22.516613 22256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "c5592c34e2e64e61bf4d139ae4be4a7c"
format_stamp: "Formatted at 2026-06-20 14:53:22 on dist-test-slave-3mqx"
server_key: "409a92177e13f583a7ff6a0efafc369b"
server_key_iv: "f8dcd32b0d4647c699f5cb6af8d44b0d"
server_key_version: "encryptionkey@0"
I20260620 14:53:22.516860 22256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:22.541764 22256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:22.546677 22256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:22.547026 22256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:22.548195 22256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:22.549687 22256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:22.549819 22256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:22.549988 22256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:22.550069 22256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:22.589538 22256 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:37113
I20260620 14:53:22.589646 22383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:37113 every 8 connection(s)
I20260620 14:53:22.591020 22256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:22.596096 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 22256
I20260620 14:53:22.596369 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:22.597168 22143 external_mini_cluster.cc:1428] Setting key 6ab0b83d5439dfa98dd54024d0d61cb1
Jun 20 14:53:22 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967202, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:53:22.615994 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:22.594219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:44989 (local address 127.21.159.254:38089)
0620 14:53:22.594715 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:22.594720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:22.596844 (+ 2124us) server_negotiation.cc:407] Connection header received
0620 14:53:22.598507 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:22.598511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:22.598581 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:22.598688 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:22.600472 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:22.601763 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:22.603737 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:22.604268 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:22.604393 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:22.609154 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:22.609159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:22.609163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:22.609204 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:22.612337 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:22.613274 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:22.613279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:22.613283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:22.613433 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:22.613925 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:22.613931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:22.613934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:22.614296 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:22.614470 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:22.615356 (+ 886us) server_negotiation.cc:299] Negotiation successful
0620 14:53:22.615615 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:53:22.617277 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:22.594881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38089 (local address 127.21.159.193:44989)
0620 14:53:22.596285 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:22.596355 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:22.598177 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:22.599036 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:22.599075 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:22.599663 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:22.600272 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:22.600289 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:22.602017 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:22.602022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:22.603565 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:22.603579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:22.604603 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:22.604614 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:22.604979 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:22.605848 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:22.605883 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:22.608824 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:22.612566 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:22.612575 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:22.612591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:22.613111 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:22.613567 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:22.613572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:22.613575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:22.613782 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:22.614448 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:22.614457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:22.614845 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 14:53:22.615233 (+ 388us) client_negotiation.cc:240] Negotiation successful
0620 14:53:22.615820 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":386,"threads_started":1}
I20260620 14:53:22.619174 22384 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38089
I20260620 14:53:22.619535 22384 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:22.620841 22384 heartbeater.cc:511] Master 127.21.159.254:38089 requested a full tablet report, sending...
I20260620 14:53:22.623231 22195 ts_manager.cc:194] Registered new tserver with Master: c5592c34e2e64e61bf4d139ae4be4a7c (127.21.159.193:37113)
I20260620 14:53:22.625046 22195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:44989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:22.642457 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38089
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:22.996481 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:22.996868 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:22.997043 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:23.004884 22390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:23.005064 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:23.005252 22390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:23.016185 22390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38089
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:23.017648 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:23.019122 22390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:23.034142 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:23.034423 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:23.034453 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:23.035921 22390 server_base.cc:1034] running on GCE node
I20260620 14:53:23.036629 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:23.037708 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:23.038951 22390 hybrid_clock.cc:648] HybridClock initialized: now 1781967203038851 us; error 107 us; skew 500 ppm
I20260620 14:53:23.039338 22390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:23.042783 22390 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:23.044791 22390 webserver.cc:460] Webserver started at http://127.21.159.194:36157/ using document root <none> and password file <none>
I20260620 14:53:23.045568 22390 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:23.045729 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:23.046082 22390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:23.049353 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "fa84ea6f837641f7b3b1281ed2e19a26"
format_stamp: "Formatted at 2026-06-20 14:53:23 on dist-test-slave-3mqx"
server_key: "06431b3e47d50f647081c9955ee959e2"
server_key_iv: "91557b327e4476d49908d2e1fdd833a0"
server_key_version: "encryptionkey@0"
I20260620 14:53:23.050271 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "fa84ea6f837641f7b3b1281ed2e19a26"
format_stamp: "Formatted at 2026-06-20 14:53:23 on dist-test-slave-3mqx"
server_key: "06431b3e47d50f647081c9955ee959e2"
server_key_iv: "91557b327e4476d49908d2e1fdd833a0"
server_key_version: "encryptionkey@0"
I20260620 14:53:23.056284 22390 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 14:53:23.060693 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:23.062371 22390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:53:23.062618 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "fa84ea6f837641f7b3b1281ed2e19a26"
format_stamp: "Formatted at 2026-06-20 14:53:23 on dist-test-slave-3mqx"
server_key: "06431b3e47d50f647081c9955ee959e2"
server_key_iv: "91557b327e4476d49908d2e1fdd833a0"
server_key_version: "encryptionkey@0"
I20260620 14:53:23.062871 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:23.083666 22390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:23.088645 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:23.089006 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:23.090109 22390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:23.091655 22390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:23.091764 22390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:23.091915 22390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:23.091991 22390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:23.130575 22390 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:43527
I20260620 14:53:23.130613 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:43527 every 8 connection(s)
I20260620 14:53:23.132066 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:23.141258 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 22390
I20260620 14:53:23.141490 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:23.141923 22143 external_mini_cluster.cc:1428] Setting key 2c6931146dff254e5aabe3bf74c373c8
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:23.154796 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.135159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:60861 (local address 127.21.159.254:38089)
0620 14:53:23.135609 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:23.135614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:23.136655 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 14:53:23.138585 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:23.138591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:23.138686 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:23.138816 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:23.140487 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.141185 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.142995 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.143660 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.143803 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.148091 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:23.148097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:23.148100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:23.148138 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:23.151456 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.152298 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.152304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.152312 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.152451 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.152937 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.152947 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.152949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.153220 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:23.153386 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:23.154159 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 14:53:23.154413 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":299,"thread_start_us":198,"threads_started":1}
I20260620 14:53:23.155932 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.135736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38089 (local address 127.21.159.194:60861)
0620 14:53:23.136452 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:23.136498 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:23.138273 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:23.139138 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:23.139148 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:23.139684 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:23.140281 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:23.140301 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.141375 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.141380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:23.142819 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:23.142832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.143924 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.143934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:23.144193 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.144957 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:23.145004 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:23.147766 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:23.151644 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:23.151656 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:23.151677 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:23.152141 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:23.152579 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:23.152585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:23.152589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:23.152803 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:23.153361 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:23.153370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:23.153759 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 14:53:23.154064 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 14:53:23.154687 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:53:23.157722 22518 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38089
I20260620 14:53:23.158100 22518 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:23.158962 22518 heartbeater.cc:511] Master 127.21.159.254:38089 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:53:23.160748 22195 ts_manager.cc:194] Registered new tserver with Master: fa84ea6f837641f7b3b1281ed2e19a26 (127.21.159.194:43527)
I20260620 14:53:23.161612 22195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:60861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:23.183562 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38089
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:23.539762 22524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:23.540082 22524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:23.540249 22524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:23.548148 22524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:23.548281 22524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:23.548498 22524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:23.558468 22524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38089
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:23.559973 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:23.561484 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:23.576766 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:23.576766 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:23.577010 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:23.577870 22524 server_base.cc:1034] running on GCE node
I20260620 14:53:23.578580 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:23.579882 22524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:23.581190 22524 hybrid_clock.cc:648] HybridClock initialized: now 1781967203581137 us; error 51 us; skew 500 ppm
I20260620 14:53:23.581622 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:23.585213 22524 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:23.586956 22524 webserver.cc:460] Webserver started at http://127.21.159.195:40581/ using document root <none> and password file <none>
I20260620 14:53:23.587846 22524 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:23.587955 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:23.588231 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:23.591728 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "f4350919d1a64a98a4da4985d84ad8b9"
format_stamp: "Formatted at 2026-06-20 14:53:23 on dist-test-slave-3mqx"
server_key: "9a4d19de76e70fa943d70b3dc2ae3462"
server_key_iv: "cec210efe8b955bac5f22ff81c48cbd8"
server_key_version: "encryptionkey@0"
I20260620 14:53:23.592612 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "f4350919d1a64a98a4da4985d84ad8b9"
format_stamp: "Formatted at 2026-06-20 14:53:23 on dist-test-slave-3mqx"
server_key: "9a4d19de76e70fa943d70b3dc2ae3462"
server_key_iv: "cec210efe8b955bac5f22ff81c48cbd8"
server_key_version: "encryptionkey@0"
I20260620 14:53:23.599498 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:53:23.603986 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:23.605839 22524 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:53:23.606092 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "f4350919d1a64a98a4da4985d84ad8b9"
format_stamp: "Formatted at 2026-06-20 14:53:23 on dist-test-slave-3mqx"
server_key: "9a4d19de76e70fa943d70b3dc2ae3462"
server_key_iv: "cec210efe8b955bac5f22ff81c48cbd8"
server_key_version: "encryptionkey@0"
I20260620 14:53:23.606357 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:23.626402 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:23.628278 22384 heartbeater.cc:503] Master 127.21.159.254:38089 was elected leader, sending a full tablet report...
I20260620 14:53:23.632087 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:23.632351 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:23.633882 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:23.635550 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:23.635627 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:23.635777 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:23.635828 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:23.672817 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:35143
I20260620 14:53:23.672876 22651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:35143 every 8 connection(s)
I20260620 14:53:23.674351 22524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:23.683851 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 22524
I20260620 14:53:23.684055 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:23.684609 22143 external_mini_cluster.cc:1428] Setting key b06733f45ccd258369fd2117e8841e48
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:23.698217 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.677343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:48011 (local address 127.21.159.254:38089)
0620 14:53:23.677880 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:23.677887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:23.678917 (+ 1030us) server_negotiation.cc:407] Connection header received
0620 14:53:23.680866 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:23.680870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:23.681054 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:23.681157 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:23.683512 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.684820 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.686870 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.687439 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.687540 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.691740 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:23.691745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:23.691749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:23.691786 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:23.694775 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.695713 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.695719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.695728 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.695854 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.696340 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.696350 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.696353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.696646 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:23.696817 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:23.697596 (+ 779us) server_negotiation.cc:299] Negotiation successful
0620 14:53:23.697823 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":291,"threads_started":1}
I20260620 14:53:23.699390 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.677969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38089 (local address 127.21.159.195:48011)
0620 14:53:23.678734 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:23.678780 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:23.680536 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:23.681437 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:23.681447 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:23.682146 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:23.683306 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:23.683328 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.684999 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.685003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:23.686700 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:23.686713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.687712 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.687720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:23.688035 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.688991 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:23.689021 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:23.691465 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:23.694985 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:23.694994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:23.695010 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:23.695557 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:23.695981 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:23.695986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:23.695990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:23.696205 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:23.696789 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:23.696797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:23.697192 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 14:53:23.697483 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 14:53:23.698080 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":281,"threads_started":1}
I20260620 14:53:23.701092 22652 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38089
I20260620 14:53:23.701434 22652 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:23.702216 22652 heartbeater.cc:511] Master 127.21.159.254:38089 requested a full tablet report, sending...
I20260620 14:53:23.703863 22195 ts_manager.cc:194] Registered new tserver with Master: f4350919d1a64a98a4da4985d84ad8b9 (127.21.159.195:35143)
I20260620 14:53:23.704711 22195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:48011
I20260620 14:53:23.714269 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:23.763283 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.745435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.21.159.193:37113)
0620 14:53:23.746575 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:23.746587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:23.746629 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:53:23.746925 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:23.746932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:23.747239 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:23.747441 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:23.749004 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.750049 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.751516 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.752129 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.752309 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.756768 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:23.756782 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:23.756798 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:23.756844 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:23.760146 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.760973 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.760982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.760984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.761124 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.761590 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.761596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.761599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.761966 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:23.762289 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:23.762615 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:53:23.762899 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":771,"threads_started":1}
W20260620 14:53:23.764631 22299 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43662
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:23.812265 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.796781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.21.159.193:37113)
0620 14:53:23.797022 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:23.797054 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:23.797194 (+ 140us) server_negotiation.cc:407] Connection header received
0620 14:53:23.797439 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:23.797444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:23.797554 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:23.797685 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:23.799215 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.799938 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.801254 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.801701 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.801885 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.806392 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:23.806397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:23.806400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:23.806441 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:23.809243 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.810049 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.810056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.810061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.810208 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.810721 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.810727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.810732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.811012 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:23.811317 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:23.811680 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:53:23.811913 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 14:53:23.814023 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 22256
I20260620 14:53:23.829210 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.815203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.21.159.193:37113)
0620 14:53:23.815419 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:23.815424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:23.815860 (+ 436us) server_negotiation.cc:407] Connection header received
0620 14:53:23.815976 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:23.815980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:23.816041 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:23.816145 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:23.817970 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.818734 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.820271 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.820796 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.820896 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.823376 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:23.823381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:23.823385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:23.823421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:23.826241 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.827215 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.827220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.827222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.827367 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.827758 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.827762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.827764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.828041 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:23.828240 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:23.828553 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:53:23.828898 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:53:23.830227 22379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:23.917981 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 22256
I20260620 14:53:23.934860 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 22390
Jun 20 14:53:23 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:53:23.955600 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:23.935905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.21.159.194:43527)
0620 14:53:23.936469 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:23.936477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:23.936503 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:53:23.936562 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:23.936568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:23.936789 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:23.936922 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:23.938628 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.939832 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.941615 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:23.942224 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:23.942341 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:23.947866 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:23.947884 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:23.947900 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:23.947945 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:23.952042 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.953157 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.953170 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.953173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.953360 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:23.953832 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:23.953835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:23.953837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:23.954189 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:23.954449 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:23.954858 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 14:53:23.955147 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":307,"threads_started":1}
I20260620 14:53:23.957165 22513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:24.044257 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 22390
I20260620 14:53:24.061790 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 22524
Jun 20 14:53:24 dist-test-slave-3mqx krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:53:24.080302 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:24.063117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.21.159.195:35143)
0620 14:53:24.063828 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:24.063836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:24.063869 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:53:24.063936 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:24.063943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:24.064161 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:24.064317 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:24.065675 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:24.066821 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:24.068334 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:24.068947 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:24.069072 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:24.073929 (+ 4857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:24.073943 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:24.073959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:24.074000 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:24.077148 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:24.077992 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:24.078000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:24.078003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:24.078152 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:24.078566 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:24.078573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:24.078576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:24.078953 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:24.079343 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:24.079655 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:53:24.079933 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":365,"threads_started":1}
I20260620 14:53:24.082046 22647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:24.170434 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 22524
I20260620 14:53:24.188279 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 22163
I20260620 14:53:24.189594 22225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:24.293730 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 22163
2026-06-20T14:53:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4120 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:24 dist-test-slave-3mqx krb5kdc[22674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:24 dist-test-slave-3mqx krb5kdc[22674](info): set up 2 sockets
Jun 20 14:53:24 dist-test-slave-3mqx krb5kdc[22674](info): commencing operation
krb5kdc: starting...
W20260620 14:53:25.443611 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:25 dist-test-slave-3mqx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:25.639940 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:43553
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:25.978653 22687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:25.979050 22687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:25.979241 22687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:25.986793 22687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:25.986980 22687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:25.987128 22687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:25.987231 22687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:25.987295 22687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:25.997666 22687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:43553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:43553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:25.999336 22687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:26.000762 22687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:26.014199 22693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:26.014285 22695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:26.014262 22692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:26.015195 22687 server_base.cc:1034] running on GCE node
I20260620 14:53:26.016018 22687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:26.017590 22687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:26.018826 22687 hybrid_clock.cc:648] HybridClock initialized: now 1781967206018767 us; error 68 us; skew 500 ppm
I20260620 14:53:26.019309 22687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:26 dist-test-slave-3mqx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967206, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:26.022806 22687 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:26.024592 22687 webserver.cc:460] Webserver started at http://127.21.159.254:33323/ using document root <none> and password file <none>
I20260620 14:53:26.025326 22687 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:26.025439 22687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:26.025705 22687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:26.029057 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "b56d5c7be57e496295bd28cf909f7747"
format_stamp: "Formatted at 2026-06-20 14:53:26 on dist-test-slave-3mqx"
server_key: "a28598ed4874002b2e0805e4306a2955"
server_key_iv: "16d5ccbccd1b6c05f32bda4247a5c5e8"
server_key_version: "encryptionkey@0"
I20260620 14:53:26.029857 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "b56d5c7be57e496295bd28cf909f7747"
format_stamp: "Formatted at 2026-06-20 14:53:26 on dist-test-slave-3mqx"
server_key: "a28598ed4874002b2e0805e4306a2955"
server_key_iv: "16d5ccbccd1b6c05f32bda4247a5c5e8"
server_key_version: "encryptionkey@0"
I20260620 14:53:26.035969 22687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:53:26.040335 22702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:26.042140 22687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:53:26.042335 22687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "b56d5c7be57e496295bd28cf909f7747"
format_stamp: "Formatted at 2026-06-20 14:53:26 on dist-test-slave-3mqx"
server_key: "a28598ed4874002b2e0805e4306a2955"
server_key_iv: "16d5ccbccd1b6c05f32bda4247a5c5e8"
server_key_version: "encryptionkey@0"
I20260620 14:53:26.042534 22687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:26.057463 22687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:26.062323 22687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:26.062589 22687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:26.083143 22687 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:43553
I20260620 14:53:26.083266 22753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:43553 every 8 connection(s)
I20260620 14:53:26.084697 22687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:26.088989 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 22687
I20260620 14:53:26.089253 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:26.089391 22754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:26.090197 22143 external_mini_cluster.cc:1428] Setting key 88afb2c7625e2a0104222fce1a40037f
I20260620 14:53:26.103864 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747: Bootstrap starting.
Jun 20 14:53:26 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:26.107342 22754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:26.108512 22754 log.cc:824] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:26.111797 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747: No bootstrap required, opened a new log
I20260620 14:53:26.114040 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:26.092566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.21.159.254:43553)
0620 14:53:26.093536 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:26.093549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:26.093597 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:53:26.095429 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:26.095457 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:26.095950 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:26.096433 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:26.098048 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:26.099471 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:26.100815 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:26.101616 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:26.101864 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:26.106138 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:26.106148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:26.106164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:26.106210 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:26.109609 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:26.110392 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:26.110405 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:26.110418 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:26.110587 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:26.110998 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:26.111004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:26.111007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:26.111616 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:26.111935 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:26.112471 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 14:53:26.112924 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":497,"threads_started":1}
I20260620 14:53:26.120662 22754 raft_consensus.cc:348] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56d5c7be57e496295bd28cf909f7747" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43553 } }
I20260620 14:53:26.121239 22754 raft_consensus.cc:374] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:26.121359 22754 raft_consensus.cc:729] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56d5c7be57e496295bd28cf909f7747, State: Initialized, Role: FOLLOWER
I20260620 14:53:26.122025 22754 consensus_queue.cc:260] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56d5c7be57e496295bd28cf909f7747" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43553 } }
I20260620 14:53:26.122277 22754 raft_consensus.cc:388] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:26.122435 22754 raft_consensus.cc:482] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:26.122619 22754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:26.124995 22754 raft_consensus.cc:504] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56d5c7be57e496295bd28cf909f7747" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43553 } }
I20260620 14:53:26.125553 22754 leader_election.cc:302] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b56d5c7be57e496295bd28cf909f7747; no voters:
I20260620 14:53:26.126256 22754 leader_election.cc:288] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:26.126379 22759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:26.127241 22759 raft_consensus.cc:686] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [term 1 LEADER]: Becoming Leader. State: Replica: b56d5c7be57e496295bd28cf909f7747, State: Running, Role: LEADER
I20260620 14:53:26.127825 22759 consensus_queue.cc:237] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56d5c7be57e496295bd28cf909f7747" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43553 } }
I20260620 14:53:26.128759 22754 sys_catalog.cc:564] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:26.131114 22760 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b56d5c7be57e496295bd28cf909f7747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56d5c7be57e496295bd28cf909f7747" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43553 } } }
I20260620 14:53:26.131875 22760 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:26.131378 22761 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b56d5c7be57e496295bd28cf909f7747. Latest consensus state: current_term: 1 leader_uuid: "b56d5c7be57e496295bd28cf909f7747" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56d5c7be57e496295bd28cf909f7747" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43553 } } }
I20260620 14:53:26.133257 22761 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:26.135120 22768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:26.140074 22768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:26.150368 22768 catalog_manager.cc:1347] Generated new cluster ID: b24ed58d9f1d46aa8df044b68e88380f
I20260620 14:53:26.150523 22768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:26.162369 22768 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:26.163900 22768 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:26.174669 22768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b56d5c7be57e496295bd28cf909f7747: Generated new TSK 0
I20260620 14:53:26.175695 22768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:26.227422 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43553
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:26.577435 22780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:26.577816 22780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:26.577982 22780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:26.585595 22780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:26.585795 22780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:26.585999 22780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:26.596349 22780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:26.597831 22780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:26.599382 22780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:26.614460 22785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:26.619602 22786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:26.619807 22788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:26.620694 22780 server_base.cc:1034] running on GCE node
I20260620 14:53:26.621483 22780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:26.622613 22780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:26.623871 22780 hybrid_clock.cc:648] HybridClock initialized: now 1781967206623822 us; error 43 us; skew 500 ppm
I20260620 14:53:26.624244 22780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:26 dist-test-slave-3mqx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967206, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:26.627827 22780 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:26.629676 22780 webserver.cc:460] Webserver started at http://127.21.159.193:43067/ using document root <none> and password file <none>
I20260620 14:53:26.630468 22780 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:26.630631 22780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:26.630963 22780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:26.634440 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "11cb3bbfbab44c58bcc610c54f26a550"
format_stamp: "Formatted at 2026-06-20 14:53:26 on dist-test-slave-3mqx"
server_key: "7171935fbb965e41392eb1ac45429c0e"
server_key_iv: "1363fc96a68b10b91b51a5e7cc8800a8"
server_key_version: "encryptionkey@0"
I20260620 14:53:26.635439 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "11cb3bbfbab44c58bcc610c54f26a550"
format_stamp: "Formatted at 2026-06-20 14:53:26 on dist-test-slave-3mqx"
server_key: "7171935fbb965e41392eb1ac45429c0e"
server_key_iv: "1363fc96a68b10b91b51a5e7cc8800a8"
server_key_version: "encryptionkey@0"
I20260620 14:53:26.641301 22780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 14:53:26.645426 22795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:26.647234 22780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:26.647444 22780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "11cb3bbfbab44c58bcc610c54f26a550"
format_stamp: "Formatted at 2026-06-20 14:53:26 on dist-test-slave-3mqx"
server_key: "7171935fbb965e41392eb1ac45429c0e"
server_key_iv: "1363fc96a68b10b91b51a5e7cc8800a8"
server_key_version: "encryptionkey@0"
I20260620 14:53:26.647677 22780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:26.666200 22780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:26.671217 22780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:26.671553 22780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:26.672616 22780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:26.674173 22780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:26.674286 22780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:26.674444 22780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:26.674522 22780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:26.710661 22780 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:39845
I20260620 14:53:26.710767 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:39845 every 8 connection(s)
I20260620 14:53:26.712059 22780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:26.718508 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 22780
I20260620 14:53:26.718770 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:26.719501 22143 external_mini_cluster.cc:1428] Setting key 5b5bb97591bc746b13049b866f68b624
Jun 20 14:53:26 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967206, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:26.737735 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:26.714972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:44927 (local address 127.21.159.254:43553)
0620 14:53:26.715511 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:26.715515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:26.716517 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 14:53:26.718459 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:26.718465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:26.718581 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:26.718721 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:26.720638 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:26.721721 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:26.724686 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:26.725138 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:26.725256 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:26.730141 (+ 4885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:26.730152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:26.730160 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:26.730212 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:26.734115 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:26.735106 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:26.735115 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:26.735121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:26.735289 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:26.735797 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:26.735802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:26.735806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:26.736099 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:26.736276 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:26.737109 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 14:53:26.737359 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":330,"threads_started":1}
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:53:26.738895 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:26.715572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:43553 (local address 127.21.159.193:44927)
0620 14:53:26.716326 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:26.716367 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:26.718145 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:26.719001 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:26.719011 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:26.719712 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:26.720426 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:26.720452 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:26.721906 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:26.721913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:26.724452 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:26.724479 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:26.725425 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:26.725436 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:26.725811 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:26.726781 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:26.726820 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:26.729776 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:26.734332 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:26.734341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:26.734359 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:26.734889 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:26.735425 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:26.735428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:26.735430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:26.735653 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:26.736258 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:26.736268 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:26.736674 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 14:53:26.736982 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 14:53:26.737620 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":288,"threads_started":1}
I20260620 14:53:26.740809 22908 heartbeater.cc:347] Connected to a master server at 127.21.159.254:43553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:53:26.741389 22908 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:26.742311 22908 heartbeater.cc:511] Master 127.21.159.254:43553 requested a full tablet report, sending...
I20260620 14:53:26.744941 22719 ts_manager.cc:194] Registered new tserver with Master: 11cb3bbfbab44c58bcc610c54f26a550 (127.21.159.193:39845)
I20260620 14:53:26.747229 22719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:44927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:26.767428 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43553
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:27.109508 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:27.109827 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:27.109926 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:27.117584 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:27.117718 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:27.117837 22914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:27.128057 22914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:27.129487 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:27.130913 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:27.145388 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:27.145596 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:27.145841 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:27.146481 22914 server_base.cc:1034] running on GCE node
I20260620 14:53:27.147305 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:27.148451 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:27.149734 22914 hybrid_clock.cc:648] HybridClock initialized: now 1781967207149678 us; error 55 us; skew 500 ppm
I20260620 14:53:27.150115 22914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:27 dist-test-slave-3mqx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:27.153995 22914 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:27.155895 22914 webserver.cc:460] Webserver started at http://127.21.159.194:34515/ using document root <none> and password file <none>
I20260620 14:53:27.156675 22914 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:27.156843 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:27.157202 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:27.160368 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "2df76a5ca13f4a34a4ad898b8b65115e"
format_stamp: "Formatted at 2026-06-20 14:53:27 on dist-test-slave-3mqx"
server_key: "b1bc01c5d09054c8ffc8d90418d1d4bd"
server_key_iv: "4276c8bcb4529aae0a1e48ed047cf80f"
server_key_version: "encryptionkey@0"
I20260620 14:53:27.161201 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "2df76a5ca13f4a34a4ad898b8b65115e"
format_stamp: "Formatted at 2026-06-20 14:53:27 on dist-test-slave-3mqx"
server_key: "b1bc01c5d09054c8ffc8d90418d1d4bd"
server_key_iv: "4276c8bcb4529aae0a1e48ed047cf80f"
server_key_version: "encryptionkey@0"
I20260620 14:53:27.166879 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 14:53:27.170733 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:27.172343 22914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:27.172554 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "2df76a5ca13f4a34a4ad898b8b65115e"
format_stamp: "Formatted at 2026-06-20 14:53:27 on dist-test-slave-3mqx"
server_key: "b1bc01c5d09054c8ffc8d90418d1d4bd"
server_key_iv: "4276c8bcb4529aae0a1e48ed047cf80f"
server_key_version: "encryptionkey@0"
I20260620 14:53:27.172789 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:27.186244 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:27.190673 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:27.191027 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:27.192081 22914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:27.193549 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:27.193666 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:27.193821 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:27.193902 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:27.228941 22914 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:43541
I20260620 14:53:27.228976 23041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:43541 every 8 connection(s)
I20260620 14:53:27.230456 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:27.236454 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 22914
I20260620 14:53:27.236711 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:27.237331 22143 external_mini_cluster.cc:1428] Setting key 9b962beffaba7ee2d5e2f32e32fbfe97
Jun 20 14:53:27 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:27.253535 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.233357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:38195 (local address 127.21.159.254:43553)
0620 14:53:27.233575 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:27.233581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:27.234981 (+ 1400us) server_negotiation.cc:407] Connection header received
0620 14:53:27.237216 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:27.237223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:27.237324 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:27.237457 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:27.239463 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.240312 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.242015 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.242440 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.242586 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.246940 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:27.246946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:27.246949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:27.246984 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:27.249796 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:27.250692 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:27.250698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:27.250707 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:27.250840 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:27.251487 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:27.251491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:27.251493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:27.251805 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:27.251968 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:27.252951 (+ 983us) server_negotiation.cc:299] Negotiation successful
0620 14:53:27.253190 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:53:27.254639 23044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.234028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:43553 (local address 127.21.159.194:38195)
0620 14:53:27.234795 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:27.234838 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:27.236884 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:27.237733 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:27.237742 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:27.238468 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:27.239171 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.239194 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.240496 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.240502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.241840 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.241853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.242706 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.242714 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.242958 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.243806 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:27.243842 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:27.246677 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:27.249989 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:27.249999 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:27.250021 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:27.250530 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:27.250971 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:27.250976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:27.250980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:27.251347 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:27.251945 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:27.251955 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:27.252500 (+ 545us) client_negotiation.cc:769] Sending connection context
0620 14:53:27.252850 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 14:53:27.253453 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":288,"threads_started":1}
I20260620 14:53:27.256314 23042 heartbeater.cc:347] Connected to a master server at 127.21.159.254:43553
I20260620 14:53:27.256737 23042 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:53:27.257745 23042 heartbeater.cc:511] Master 127.21.159.254:43553 requested a full tablet report, sending...
I20260620 14:53:27.259505 22719 ts_manager.cc:194] Registered new tserver with Master: 2df76a5ca13f4a34a4ad898b8b65115e (127.21.159.194:43541)
I20260620 14:53:27.260382 22719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:38195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:27.282248 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43553
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:27.631665 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:27.631966 23047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:27.632176 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:27.640164 23047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:27.640285 23047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:27.640463 23047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:27.650779 23047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43553
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:27.652343 23047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:27.653807 23047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:27.668473 23053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:27.668495 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:27.669080 23055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:27.669483 23047 server_base.cc:1034] running on GCE node
I20260620 14:53:27.670157 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:27.671373 23047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:27.672600 23047 hybrid_clock.cc:648] HybridClock initialized: now 1781967207672543 us; error 52 us; skew 500 ppm
I20260620 14:53:27.672961 23047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:27 dist-test-slave-3mqx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:27.676685 23047 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:27.678646 23047 webserver.cc:460] Webserver started at http://127.21.159.195:42859/ using document root <none> and password file <none>
I20260620 14:53:27.679541 23047 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:27.679698 23047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:27.680022 23047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:27.683197 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "c40b3bc387234c888e963f96923114cf"
format_stamp: "Formatted at 2026-06-20 14:53:27 on dist-test-slave-3mqx"
server_key: "51727015d1d290497dd05ce8fe837e68"
server_key_iv: "2d97a013bdb634d7f65974b6288c20f0"
server_key_version: "encryptionkey@0"
I20260620 14:53:27.684058 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "c40b3bc387234c888e963f96923114cf"
format_stamp: "Formatted at 2026-06-20 14:53:27 on dist-test-slave-3mqx"
server_key: "51727015d1d290497dd05ce8fe837e68"
server_key_iv: "2d97a013bdb634d7f65974b6288c20f0"
server_key_version: "encryptionkey@0"
I20260620 14:53:27.689870 23047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:53:27.694182 23062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:27.695953 23047 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:53:27.696169 23047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "c40b3bc387234c888e963f96923114cf"
format_stamp: "Formatted at 2026-06-20 14:53:27 on dist-test-slave-3mqx"
server_key: "51727015d1d290497dd05ce8fe837e68"
server_key_iv: "2d97a013bdb634d7f65974b6288c20f0"
server_key_version: "encryptionkey@0"
I20260620 14:53:27.696408 23047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:27.725260 23047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:27.730268 23047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:27.730588 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:27.731729 23047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:27.733429 23047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:27.733538 23047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:27.733678 23047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:27.733757 23047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:27.751425 22908 heartbeater.cc:503] Master 127.21.159.254:43553 was elected leader, sending a full tablet report...
I20260620 14:53:27.769797 23047 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:35053
I20260620 14:53:27.769836 23174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:35053 every 8 connection(s)
I20260620 14:53:27.771334 23047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:27.772943 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 23047
I20260620 14:53:27.773182 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:27.773643 22143 external_mini_cluster.cc:1428] Setting key 7b585a3ffbf8ba6357fa76c2d4a95442
Jun 20 14:53:27 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967207, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:27.795037 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.774294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:36235 (local address 127.21.159.254:43553)
0620 14:53:27.774765 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:27.774787 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:27.776330 (+ 1543us) server_negotiation.cc:407] Connection header received
0620 14:53:27.778880 (+ 2550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:27.778884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:27.778951 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:27.779045 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:27.780917 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.781702 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.783669 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.784115 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.784222 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.788145 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:27.788151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:27.788154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:27.788188 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:27.791919 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:27.792775 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:27.792780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:27.792782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:27.792915 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:27.793377 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:27.793381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:27.793387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:27.793667 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:27.793802 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:27.794570 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 14:53:27.794745 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260620 14:53:27.796188 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.775115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:43553 (local address 127.21.159.195:36235)
0620 14:53:27.776129 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:27.776199 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:27.778582 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:27.779443 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:27.779459 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:27.780090 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:27.780746 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.780768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.781895 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.781899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.783514 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.783532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.784371 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.784378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.784644 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.785377 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:27.785403 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:27.787862 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:27.792086 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:27.792095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:27.792113 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:27.792615 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:27.793021 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:27.793024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:27.793026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:27.793242 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:27.793792 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:27.793798 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:27.794149 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 14:53:27.794417 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 14:53:27.794868 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":367,"threads_started":1}
I20260620 14:53:27.797894 23175 heartbeater.cc:347] Connected to a master server at 127.21.159.254:43553
I20260620 14:53:27.798243 23175 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:27.799016 23175 heartbeater.cc:511] Master 127.21.159.254:43553 requested a full tablet report, sending...
I20260620 14:53:27.800832 22719 ts_manager.cc:194] Registered new tserver with Master: c40b3bc387234c888e963f96923114cf (127.21.159.195:35053)
I20260620 14:53:27.801800 22719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:36235
I20260620 14:53:27.803741 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:27.825139 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.810962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.21.159.254:43553)
0620 14:53:27.811164 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:27.811169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:27.811782 (+ 613us) server_negotiation.cc:407] Connection header received
0620 14:53:27.812002 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:27.812008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:27.812087 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:27.812281 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:27.813776 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.814539 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.816000 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.816426 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.816570 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.819007 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:27.819014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:27.819019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:27.819122 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:27.822232 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:27.822989 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:27.822995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:27.822998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:27.823211 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:27.823608 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:27.823616 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:27.823619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:27.823955 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:27.824228 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:27.824561 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:53:27.824797 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:53:27.832959 22719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:27.841723 22719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:27.871025 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.858450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.21.159.194:43541)
0620 14:53:27.859779 (+ 1329us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:27.859791 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:27.862353 (+ 2562us) server_negotiation.cc:407] Connection header received
0620 14:53:27.862680 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:27.862688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:27.863003 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:27.863249 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:27.863837 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.865935 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.868287 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.869592 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.869776 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.869954 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:27.870645 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 14:53:27.870795 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":1095,"threads_started":1}
I20260620 14:53:27.871233 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.857820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.21.159.195:35053)
0620 14:53:27.859342 (+ 1522us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:27.859356 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:27.860430 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 14:53:27.860895 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:27.860903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:27.861244 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:27.861511 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:27.863000 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.864668 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.867852 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.869483 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.869681 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.869878 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:27.870640 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 14:53:27.870820 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1298,"thread_start_us":1117,"threads_started":1}
I20260620 14:53:27.871357 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.860977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:39845 (local address 127.0.0.1:49472)
0620 14:53:27.862241 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:27.862260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:27.862371 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:27.863667 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:27.863677 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:27.863704 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:27.864027 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.864037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.866129 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.866134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.868380 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.868393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.870306 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.870313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.870598 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.870637 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:53:27.870706 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:53:27.870896 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"mutex_wait_us":71,"thread_start_us":1009,"threads_started":1}
I20260620 14:53:27.871825 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.857908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:43541 (local address 127.0.0.1:56708)
0620 14:53:27.862231 (+ 4323us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:27.862251 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:27.862464 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:27.863213 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:27.863218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:27.863243 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:27.863611 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.863633 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.866134 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.866139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.868079 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.868094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.870104 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.870110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.870308 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.870436 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:53:27.871155 (+ 719us) client_negotiation.cc:240] Negotiation successful
0620 14:53:27.871329 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2858,"mutex_wait_us":83,"thread_start_us":2695,"threads_started":1}
I20260620 14:53:27.872005 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.857563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:35053 (local address 127.0.0.1:53948)
0620 14:53:27.860271 (+ 2708us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:27.860315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:27.860612 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:27.862241 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:27.862250 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:27.862290 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:27.862746 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.862760 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.864840 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.864847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.867619 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:27.867641 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.870042 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.870048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:27.870299 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.870400 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 14:53:27.871450 (+ 1050us) client_negotiation.cc:240] Negotiation successful
0620 14:53:27.871597 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2435,"mutex_wait_us":52,"thread_start_us":2304,"threads_started":1}
I20260620 14:53:27.872779 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:27.861259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.21.159.193:39845)
0620 14:53:27.862990 (+ 1731us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:27.862999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:27.863030 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:53:27.863193 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:27.863200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:27.863442 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:27.863654 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:27.864433 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.865966 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.868573 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:27.870067 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:27.871935 (+ 1868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:27.872346 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:27.872409 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:53:27.872543 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":1379,"threads_started":1}
I20260620 14:53:27.876395 22843 tablet_service.cc:1467] Processing CreateTablet for tablet cd3823e1d1a94924b54ea9601af4caa4 (DEFAULT_TABLE table=test-table [id=a3e0235d7e6441048f43731c1a096533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:27.876392 22977 tablet_service.cc:1467] Processing CreateTablet for tablet cd3823e1d1a94924b54ea9601af4caa4 (DEFAULT_TABLE table=test-table [id=a3e0235d7e6441048f43731c1a096533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:27.877990 22843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3823e1d1a94924b54ea9601af4caa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:27.878006 22977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3823e1d1a94924b54ea9601af4caa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:27.882107 23110 tablet_service.cc:1467] Processing CreateTablet for tablet cd3823e1d1a94924b54ea9601af4caa4 (DEFAULT_TABLE table=test-table [id=a3e0235d7e6441048f43731c1a096533]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:27.883682 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3823e1d1a94924b54ea9601af4caa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:27.894318 23194 tablet_bootstrap.cc:492] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e: Bootstrap starting.
I20260620 14:53:27.895327 23195 tablet_bootstrap.cc:492] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550: Bootstrap starting.
I20260620 14:53:27.898712 23196 tablet_bootstrap.cc:492] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf: Bootstrap starting.
I20260620 14:53:27.899717 23194 tablet_bootstrap.cc:654] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:27.901222 23194 log.cc:824] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:27.901708 23195 tablet_bootstrap.cc:654] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:27.902354 23196 tablet_bootstrap.cc:654] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:27.903228 23195 log.cc:824] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:27.903564 23196 log.cc:824] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:27.904925 23194 tablet_bootstrap.cc:492] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e: No bootstrap required, opened a new log
I20260620 14:53:27.905242 23194 ts_tablet_manager.cc:1397] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 14:53:27.906564 23195 tablet_bootstrap.cc:492] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550: No bootstrap required, opened a new log
I20260620 14:53:27.906905 23195 ts_tablet_manager.cc:1397] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 14:53:27.908819 23196 tablet_bootstrap.cc:492] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf: No bootstrap required, opened a new log
I20260620 14:53:27.909104 23196 ts_tablet_manager.cc:1397] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 14:53:27.916631 23195 raft_consensus.cc:348] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:27.917140 23195 raft_consensus.cc:374] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:27.916821 23194 raft_consensus.cc:348] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:27.917256 23195 raft_consensus.cc:729] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11cb3bbfbab44c58bcc610c54f26a550, State: Initialized, Role: FOLLOWER
I20260620 14:53:27.917295 23194 raft_consensus.cc:374] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:27.917380 23194 raft_consensus.cc:729] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df76a5ca13f4a34a4ad898b8b65115e, State: Initialized, Role: FOLLOWER
I20260620 14:53:27.917991 23195 consensus_queue.cc:260] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:27.917997 23194 consensus_queue.cc:260] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:27.919564 23042 heartbeater.cc:503] Master 127.21.159.254:43553 was elected leader, sending a full tablet report...
I20260620 14:53:27.920114 23194 ts_tablet_manager.cc:1428] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 14:53:27.920110 23196 raft_consensus.cc:348] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:27.920607 23196 raft_consensus.cc:374] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:27.920732 23196 raft_consensus.cc:729] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40b3bc387234c888e963f96923114cf, State: Initialized, Role: FOLLOWER
I20260620 14:53:27.921783 23195 ts_tablet_manager.cc:1428] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260620 14:53:27.922103 23196 consensus_queue.cc:260] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:27.924789 23175 heartbeater.cc:503] Master 127.21.159.254:43553 was elected leader, sending a full tablet report...
I20260620 14:53:27.925619 23196 ts_tablet_manager.cc:1428] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
W20260620 14:53:27.964751 22909 tablet.cc:2369] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:53:27.982817 23043 tablet.cc:2369] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:53:28.023362 23176 tablet.cc:2369] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:28.102016 23201 raft_consensus.cc:482] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:28.102372 23201 raft_consensus.cc:504] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:28.104225 23201 leader_election.cc:288] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c40b3bc387234c888e963f96923114cf (127.21.159.195:35053), 11cb3bbfbab44c58bcc610c54f26a550 (127.21.159.193:39845)
I20260620 14:53:28.114529 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.104698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:33751 (local address 127.21.159.195:35053)
0620 14:53:28.104899 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.104906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.105815 (+ 909us) server_negotiation.cc:407] Connection header received
0620 14:53:28.106069 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.106074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.106153 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.106259 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:28.107758 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.108921 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.112014 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.113307 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.113496 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.113576 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.114120 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.114260 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:53:28.114532 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.104690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:54685 (local address 127.21.159.193:39845)
0620 14:53:28.104899 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.104905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.105844 (+ 939us) server_negotiation.cc:407] Connection header received
0620 14:53:28.106028 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.106034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.106120 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.106293 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:28.107758 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.108879 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.112014 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.113307 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.113496 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.113576 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.114123 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.114260 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:53:28.114784 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.104610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:39845 (local address 127.21.159.194:54685)
0620 14:53:28.105640 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:28.105661 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:28.105846 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:28.106319 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:28.106324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:28.106354 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:28.107554 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:28.107566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.109101 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.109107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:28.111755 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:28.111775 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.113625 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.113632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:28.113857 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.113950 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 14:53:28.114059 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:53:28.114301 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":297,"thread_start_us":497,"threads_started":1}
I20260620 14:53:28.115260 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.104680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:35053 (local address 127.21.159.194:33751)
0620 14:53:28.105705 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:28.105721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:28.105897 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:28.106319 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:28.106324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:28.106384 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:28.107511 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:28.107536 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.109101 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.109107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:28.111755 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:28.111776 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.113636 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.113641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:28.113854 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.113961 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 14:53:28.114059 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:53:28.114301 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":181,"spinlock_wait_cycles":32640,"thread_start_us":621,"threads_started":1}
I20260620 14:53:28.115965 22863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3823e1d1a94924b54ea9601af4caa4" candidate_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11cb3bbfbab44c58bcc610c54f26a550" is_pre_election: true
I20260620 14:53:28.116391 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3823e1d1a94924b54ea9601af4caa4" candidate_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40b3bc387234c888e963f96923114cf" is_pre_election: true
I20260620 14:53:28.116600 22863 raft_consensus.cc:2454] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df76a5ca13f4a34a4ad898b8b65115e in term 0.
I20260620 14:53:28.116930 23130 raft_consensus.cc:2454] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df76a5ca13f4a34a4ad898b8b65115e in term 0.
I20260620 14:53:28.117324 22931 leader_election.cc:302] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11cb3bbfbab44c58bcc610c54f26a550, 2df76a5ca13f4a34a4ad898b8b65115e; no voters:
I20260620 14:53:28.117735 23201 raft_consensus.cc:2781] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:28.117887 23201 raft_consensus.cc:482] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:28.117996 23201 raft_consensus.cc:3037] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:28.120239 23201 raft_consensus.cc:504] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:28.120880 23201 leader_election.cc:288] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [CANDIDATE]: Term 1 election: Requested vote from peers c40b3bc387234c888e963f96923114cf (127.21.159.195:35053), 11cb3bbfbab44c58bcc610c54f26a550 (127.21.159.193:39845)
I20260620 14:53:28.121371 23130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3823e1d1a94924b54ea9601af4caa4" candidate_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40b3bc387234c888e963f96923114cf"
I20260620 14:53:28.121440 22863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3823e1d1a94924b54ea9601af4caa4" candidate_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11cb3bbfbab44c58bcc610c54f26a550"
I20260620 14:53:28.121575 23130 raft_consensus.cc:3037] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:28.121717 22863 raft_consensus.cc:3037] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:28.124138 22863 raft_consensus.cc:2454] T cd3823e1d1a94924b54ea9601af4caa4 P 11cb3bbfbab44c58bcc610c54f26a550 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df76a5ca13f4a34a4ad898b8b65115e in term 1.
I20260620 14:53:28.124369 23130 raft_consensus.cc:2454] T cd3823e1d1a94924b54ea9601af4caa4 P c40b3bc387234c888e963f96923114cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df76a5ca13f4a34a4ad898b8b65115e in term 1.
I20260620 14:53:28.124679 22931 leader_election.cc:302] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11cb3bbfbab44c58bcc610c54f26a550, 2df76a5ca13f4a34a4ad898b8b65115e; no voters:
I20260620 14:53:28.124984 23201 raft_consensus.cc:2781] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:28.125474 23201 raft_consensus.cc:686] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [term 1 LEADER]: Becoming Leader. State: Replica: 2df76a5ca13f4a34a4ad898b8b65115e, State: Running, Role: LEADER
I20260620 14:53:28.125962 23201 consensus_queue.cc:237] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } }
I20260620 14:53:28.130771 22719 catalog_manager.cc:5495] T cd3823e1d1a94924b54ea9601af4caa4 P 2df76a5ca13f4a34a4ad898b8b65115e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df76a5ca13f4a34a4ad898b8b65115e (127.21.159.194). New cstate: current_term: 1 leader_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c40b3bc387234c888e963f96923114cf" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df76a5ca13f4a34a4ad898b8b65115e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11cb3bbfbab44c58bcc610c54f26a550" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39845 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:28.176487 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.159208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.21.159.193:39845)
0620 14:53:28.159432 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.159441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.159860 (+ 419us) server_negotiation.cc:407] Connection header received
0620 14:53:28.160103 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.160109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.160199 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.160330 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:28.162479 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.163377 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.164833 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.165279 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.165441 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.170112 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:28.170132 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:28.170148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:28.170191 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:28.173136 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.173972 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.173981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.173984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.174146 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.174669 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.174672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.174675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.175209 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:28.175546 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.176026 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.176210 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:28.226743 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.210664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.21.159.193:39845)
0620 14:53:28.210902 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.210908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.211320 (+ 412us) server_negotiation.cc:407] Connection header received
0620 14:53:28.211463 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.211467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.211551 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.211659 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:28.213163 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.213961 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.215832 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.216377 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.216490 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.220773 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:28.220779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:28.220783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:28.220823 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:28.223733 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.224526 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.224531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.224533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.224694 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.225199 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.225203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.225205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.225483 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:28.225694 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.226257 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.226456 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:28.278668 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.261907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.21.159.193:39845)
0620 14:53:28.262212 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.262217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.262313 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:53:28.262464 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.262468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.262537 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.262633 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:28.264500 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.265224 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.266742 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.267442 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.267566 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.272135 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:28.272145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:28.272150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:28.272210 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:28.275440 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.276400 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.276405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.276407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.276554 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.277020 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.277024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.277027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.277497 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:28.277728 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.278029 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.278227 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260620 14:53:28.280620 22823 tablet_service.cc:2908] Rejecting scan request for tablet cd3823e1d1a94924b54ea9601af4caa4: Uninitialized: clean time has not yet been initialized
I20260620 14:53:28.284431 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 22780
I20260620 14:53:28.299455 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.285631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.21.159.193:39845)
0620 14:53:28.285818 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.285823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.286313 (+ 490us) server_negotiation.cc:407] Connection header received
0620 14:53:28.286466 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.286470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.286548 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.286646 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:28.288054 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.288777 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.290053 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.290508 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.290623 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.293008 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:28.293013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:28.293016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:28.293052 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:28.296249 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.297195 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.297200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.297202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.297355 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.297860 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.297864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.297866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.298216 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:28.298459 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.298859 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.299123 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:53:28.300130 22903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:28.393736 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 22780
I20260620 14:53:28.416174 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 22914
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:53:28.433211 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.417112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.21.159.194:43541)
0620 14:53:28.417596 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.417601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.417623 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:53:28.417675 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.417679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.417747 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.417883 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:28.419099 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.419863 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.421254 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.421686 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.421871 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.426137 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:28.426161 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:28.426185 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:28.426252 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:28.429866 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.430689 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.430700 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.430703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.430866 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.431487 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.431493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.431496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.431890 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:28.432219 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.432565 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.432832 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260620 14:53:28.433966 23037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:28.527475 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 22914
I20260620 14:53:28.545065 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23047
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[22674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:53:28.561319 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.545931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.21.159.195:35053)
0620 14:53:28.546132 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.546139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.546194 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:53:28.546414 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.546418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.546479 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.546565 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:28.547979 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.548771 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.550117 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.550622 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.550748 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.555150 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:28.555167 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:28.555185 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:28.555231 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:28.558133 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.559192 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.559200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.559203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.559367 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.559841 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.559846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.559849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.560262 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:28.560485 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.560848 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.561027 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:53:28.562086 23170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:28.652782 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 23047
I20260620 14:53:28.669409 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 22687
I20260620 14:53:28.670569 22749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:28.780243 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 22687
2026-06-20T14:53:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4486 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[23225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[23225](info): set up 2 sockets
Jun 20 14:53:28 dist-test-slave-3mqx krb5kdc[23225](info): commencing operation
krb5kdc: starting...
W20260620 14:53:29.919518 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:30 dist-test-slave-3mqx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:30.118537 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36537
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:36537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:30.465631 23238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:30.466032 23238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:30.466219 23238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:30.473778 23238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:30.473908 23238 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:53:30.473949 23238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:30.474009 23238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:30.474079 23238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:30.474140 23238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:30.484448 23238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:36537
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:30.485729 23238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:30.487226 23238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:30.500291 23243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.500259 23244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.500231 23246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.501488 23238 server_base.cc:1034] running on GCE node
I20260620 14:53:30.502393 23238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:30.504211 23238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:30.505534 23238 hybrid_clock.cc:648] HybridClock initialized: now 1781967210505416 us; error 49 us; skew 500 ppm
I20260620 14:53:30.505911 23238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:30 dist-test-slave-3mqx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:30.514283 23238 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:30.516180 23238 webserver.cc:460] Webserver started at http://127.21.159.254:33279/ using document root <none> and password file <none>
I20260620 14:53:30.517017 23238 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:30.517168 23238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:30.517570 23238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:30.520830 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf"
format_stamp: "Formatted at 2026-06-20 14:53:30 on dist-test-slave-3mqx"
server_key: "c6c0c83216adcad5dab8d6159d516582"
server_key_iv: "4fd0fd5792e89be4c91d04df1cc97e2a"
server_key_version: "encryptionkey@0"
I20260620 14:53:30.521677 23238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf"
format_stamp: "Formatted at 2026-06-20 14:53:30 on dist-test-slave-3mqx"
server_key: "c6c0c83216adcad5dab8d6159d516582"
server_key_iv: "4fd0fd5792e89be4c91d04df1cc97e2a"
server_key_version: "encryptionkey@0"
I20260620 14:53:30.527686 23238 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 14:53:30.531589 23253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.533284 23238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:53:30.533550 23238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf"
format_stamp: "Formatted at 2026-06-20 14:53:30 on dist-test-slave-3mqx"
server_key: "c6c0c83216adcad5dab8d6159d516582"
server_key_iv: "4fd0fd5792e89be4c91d04df1cc97e2a"
server_key_version: "encryptionkey@0"
I20260620 14:53:30.533838 23238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:30.563539 23238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:30.568547 23238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:30.568852 23238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:30.592185 23238 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:36537
I20260620 14:53:30.592202 23314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:36537 every 8 connection(s)
I20260620 14:53:30.593699 23238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:30.597944 23315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:30.599506 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 23238
I20260620 14:53:30.599756 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:30.600391 22143 external_mini_cluster.cc:1428] Setting key eceae2183c87e0fff092fc3fb77b4fa8
I20260620 14:53:30.613698 23315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf: Bootstrap starting.
Jun 20 14:53:30 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:30.617377 23315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:30.618675 23315 log.cc:824] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:30.622670 23315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf: No bootstrap required, opened a new log
I20260620 14:53:30.627835 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:30.602700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.21.159.254:36537)
0620 14:53:30.603631 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:30.603654 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:30.603721 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:53:30.605633 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:30.605676 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:30.606314 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:30.606819 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:30.608502 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.610681 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.612323 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.613486 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.613705 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:30.618303 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:30.618322 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:30.618346 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:30.618401 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:30.622412 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.623418 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.623543 (+ 125us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.623553 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.623784 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.624174 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.624178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.624183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.624964 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:30.625267 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:30.625717 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 14:53:30.626245 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":420,"threads_started":1}
I20260620 14:53:30.633055 23315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36537 } }
I20260620 14:53:30.633453 23315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:30.633534 23315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7b38ce986c437db1d83d8d7f4f9bcf, State: Initialized, Role: FOLLOWER
I20260620 14:53:30.634164 23315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36537 } }
I20260620 14:53:30.634385 23315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:30.634478 23315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:30.634609 23315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:30.637216 23315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36537 } }
I20260620 14:53:30.637741 23315 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7b38ce986c437db1d83d8d7f4f9bcf; no voters:
I20260620 14:53:30.638470 23315 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:30.638597 23320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:30.639467 23320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [term 1 LEADER]: Becoming Leader. State: Replica: 4b7b38ce986c437db1d83d8d7f4f9bcf, State: Running, Role: LEADER
I20260620 14:53:30.640051 23320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36537 } }
I20260620 14:53:30.641058 23315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:30.643530 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7b38ce986c437db1d83d8d7f4f9bcf. Latest consensus state: current_term: 1 leader_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36537 } } }
I20260620 14:53:30.644037 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:30.644692 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7b38ce986c437db1d83d8d7f4f9bcf" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36537 } } }
I20260620 14:53:30.645102 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:30.646795 23329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:30.652462 23329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:30.662783 23329 catalog_manager.cc:1347] Generated new cluster ID: 18bc1d935d414fccbd03a579493524ed
I20260620 14:53:30.662925 23329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:30.676492 23329 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:30.678639 23329 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:30.694401 23329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf: Generated new TSK 0
I20260620 14:53:30.695587 23329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:30.752316 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36537
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:31.121031 23341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:31.121392 23341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:31.121578 23341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:31.129149 23341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:31.129346 23341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:31.129562 23341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:53:31.129726 23341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:31.140488 23341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:36537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:31.141952 23341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:31.143559 23341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:31.158190 23347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:31.158617 23346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:31.158697 23349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:31.159157 23341 server_base.cc:1034] running on GCE node
I20260620 14:53:31.159850 23341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:31.160881 23341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:31.162182 23341 hybrid_clock.cc:648] HybridClock initialized: now 1781967211162088 us; error 69 us; skew 500 ppm
I20260620 14:53:31.162638 23341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:31 dist-test-slave-3mqx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967211, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:31.166283 23341 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:31.168205 23341 webserver.cc:460] Webserver started at http://127.21.159.193:39949/ using document root <none> and password file <none>
I20260620 14:53:31.169055 23341 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:31.169219 23341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:31.169606 23341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:31.173046 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "64d9b84130d6417c83f0f707b0628a9d"
format_stamp: "Formatted at 2026-06-20 14:53:31 on dist-test-slave-3mqx"
server_key: "d1f8f895f2004454ba89f935db22f726"
server_key_iv: "2578b36dc053464a396d0094aa07bba4"
server_key_version: "encryptionkey@0"
I20260620 14:53:31.174026 23341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "64d9b84130d6417c83f0f707b0628a9d"
format_stamp: "Formatted at 2026-06-20 14:53:31 on dist-test-slave-3mqx"
server_key: "d1f8f895f2004454ba89f935db22f726"
server_key_iv: "2578b36dc053464a396d0094aa07bba4"
server_key_version: "encryptionkey@0"
I20260620 14:53:31.180305 23341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 14:53:31.184326 23356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:31.186053 23341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:53:31.186301 23341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "64d9b84130d6417c83f0f707b0628a9d"
format_stamp: "Formatted at 2026-06-20 14:53:31 on dist-test-slave-3mqx"
server_key: "d1f8f895f2004454ba89f935db22f726"
server_key_iv: "2578b36dc053464a396d0094aa07bba4"
server_key_version: "encryptionkey@0"
I20260620 14:53:31.186591 23341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:31.205081 23341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:31.210014 23341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:31.210328 23341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:31.213265 23341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:31.213408 23341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:31.213554 23341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:31.213644 23341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:53:31 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967211, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:31.239194 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.214143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:51515 (local address 127.21.159.254:36537)
0620 14:53:31.214613 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:31.214620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:31.216086 (+ 1466us) server_negotiation.cc:407] Connection header received
0620 14:53:31.218228 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:31.218233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:31.218331 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:31.218466 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:31.221649 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.222670 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.224950 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.225367 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.225537 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.230717 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:31.230725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:31.230731 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:31.230787 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:31.234603 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.235841 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.235848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.235853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.236059 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.236601 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.236609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.236616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.237105 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:31.237298 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:31.238270 (+ 972us) server_negotiation.cc:299] Negotiation successful
0620 14:53:31.238604 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260620 14:53:31.240829 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.214836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36537 (local address 127.21.159.193:51515)
0620 14:53:31.215845 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:31.215901 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:31.217880 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:31.218862 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:31.218882 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:31.220151 (+ 1269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:31.221353 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.221396 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.222917 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.222926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.224751 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.224766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.225651 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.225659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.225903 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.226789 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:31.226828 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:31.230307 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:31.234814 (+ 4507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.234828 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.234855 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.235651 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.236217 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.236222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.236225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.236450 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.237285 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:31.237293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:31.237744 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 14:53:31.238101 (+ 357us) client_negotiation.cc:240] Negotiation successful
0620 14:53:31.238554 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":493,"threads_started":1}
I20260620 14:53:31.263273 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.247631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:41503 (local address 127.21.159.254:36537)
0620 14:53:31.247899 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:31.247905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:31.248770 (+ 865us) server_negotiation.cc:407] Connection header received
0620 14:53:31.248923 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:31.248932 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:31.249049 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:31.249177 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:31.250840 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.252185 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.254625 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.255190 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.255351 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.257580 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:31.257585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:31.257589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:31.257622 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:31.260232 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.261062 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.261070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.261076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.261218 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.261644 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.261652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.261658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.261956 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:31.262153 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:31.262624 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 14:53:31.262860 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:53:31.263521 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.247672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36537 (local address 127.21.159.193:41503)
0620 14:53:31.248628 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:31.248650 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:31.248807 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:31.249228 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:31.249236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:31.249956 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:31.250612 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.250624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.252379 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.252388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.254405 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.254430 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.255444 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.255452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.255638 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.256545 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:31.256564 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:31.257338 (+ 774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:31.260405 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.260411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.260416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.260911 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.261358 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.261364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.261368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.261519 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.262123 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:31.262128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:31.262337 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 14:53:31.262525 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:53:31.262896 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":646,"threads_started":1}
I20260620 14:53:31.274695 23341 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:38161
I20260620 14:53:31.274796 23476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:38161 every 8 connection(s)
I20260620 14:53:31.276161 23341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:31.278575 23477 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36537
I20260620 14:53:31.278841 23477 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:31.279634 23477 heartbeater.cc:511] Master 127.21.159.254:36537 requested a full tablet report, sending...
I20260620 14:53:31.282037 23270 ts_manager.cc:194] Registered new tserver with Master: 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161)
I20260620 14:53:31.283799 23270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:51515
I20260620 14:53:31.285290 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 23341
I20260620 14:53:31.285578 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:31.286207 22143 external_mini_cluster.cc:1428] Setting key fbd2d2bfd82a6e7e90a3d31ff108dd0c
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:31.331882 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36537
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:31.691371 23481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:31.691751 23481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:31.691905 23481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:31.699589 23481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:31.699719 23481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:31.699887 23481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:53:31.700013 23481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:31.710412 23481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:36537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:31.711946 23481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:31.713366 23481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:31.727835 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:31.727784 23489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:31.727810 23487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:31.728835 23481 server_base.cc:1034] running on GCE node
I20260620 14:53:31.729557 23481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:31.730703 23481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:31.731983 23481 hybrid_clock.cc:648] HybridClock initialized: now 1781967211731878 us; error 80 us; skew 500 ppm
I20260620 14:53:31.732349 23481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:31 dist-test-slave-3mqx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967211, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:31.735989 23481 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:31.737936 23481 webserver.cc:460] Webserver started at http://127.21.159.194:33445/ using document root <none> and password file <none>
I20260620 14:53:31.738878 23481 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:31.739022 23481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:31.739487 23481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:31.744164 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "bc583c84da1641c0b85aa6ccf16a443a"
format_stamp: "Formatted at 2026-06-20 14:53:31 on dist-test-slave-3mqx"
server_key: "2532ec6d615a17c2fc5f4c80421caeb2"
server_key_iv: "f82266c0a1d27934e77983c1ffa03681"
server_key_version: "encryptionkey@0"
I20260620 14:53:31.745365 23481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "bc583c84da1641c0b85aa6ccf16a443a"
format_stamp: "Formatted at 2026-06-20 14:53:31 on dist-test-slave-3mqx"
server_key: "2532ec6d615a17c2fc5f4c80421caeb2"
server_key_iv: "f82266c0a1d27934e77983c1ffa03681"
server_key_version: "encryptionkey@0"
I20260620 14:53:31.751484 23481 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 14:53:31.755724 23496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:31.757390 23481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:53:31.757608 23481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "bc583c84da1641c0b85aa6ccf16a443a"
format_stamp: "Formatted at 2026-06-20 14:53:31 on dist-test-slave-3mqx"
server_key: "2532ec6d615a17c2fc5f4c80421caeb2"
server_key_iv: "f82266c0a1d27934e77983c1ffa03681"
server_key_version: "encryptionkey@0"
I20260620 14:53:31.757858 23481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:31.772374 23481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:31.777300 23481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:31.777639 23481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:31.780794 23481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:31.780916 23481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:31.781087 23481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:31.781170 23481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:53:31 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967211, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:31.811012 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.781550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:51729 (local address 127.21.159.254:36537)
0620 14:53:31.782328 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:31.782333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:31.783513 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 14:53:31.786247 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:31.786255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:31.786375 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:31.786473 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:31.789811 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.791149 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.793609 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.794341 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.794542 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.801380 (+ 6838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:31.801387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:31.801392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:31.801446 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:31.805793 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.807389 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.807398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.807403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.807638 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.808406 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.808414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.808420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.808865 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:31.809069 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:31.810284 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0620 14:53:31.810571 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"spinlock_wait_cycles":5888,"thread_start_us":607,"threads_started":1}
I20260620 14:53:31.812610 23506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.782320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36537 (local address 127.21.159.194:51729)
0620 14:53:31.783311 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:31.783361 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:31.785763 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:31.787255 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:31.787266 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:31.788360 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:31.789547 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.789592 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.791350 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.791362 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.793391 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.793408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.794708 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.794720 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.795232 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.796442 (+ 1210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:31.796487 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:31.800996 (+ 4509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:31.806087 (+ 5091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.806104 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.806131 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.807151 (+ 1020us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.807817 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.807825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.807831 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.808210 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.809072 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:31.809085 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:31.809673 (+ 588us) client_negotiation.cc:769] Sending connection context
0620 14:53:31.810165 (+ 492us) client_negotiation.cc:240] Negotiation successful
0620 14:53:31.810847 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":374,"threads_started":1}
I20260620 14:53:31.835260 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.819733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:33917 (local address 127.21.159.254:36537)
0620 14:53:31.819948 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:31.819955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:31.820849 (+ 894us) server_negotiation.cc:407] Connection header received
0620 14:53:31.821070 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:31.821076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:31.821172 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:31.821281 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:31.822259 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.823018 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.825443 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.825909 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.825988 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.828786 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:31.828791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:31.828794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:31.828827 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:31.831817 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.832794 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.832802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.832806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.832979 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.833568 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.833574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.833577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.833925 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:31.834071 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:31.834718 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 14:53:31.834915 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:53:31.835843 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.819801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36537 (local address 127.21.159.194:33917)
0620 14:53:31.820729 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:31.820749 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:31.820928 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:31.821311 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:31.821316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:31.821810 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:31.822102 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.822113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.823178 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.823188 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.825223 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.825245 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.826263 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.826268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.826408 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.827287 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:31.827307 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:31.828554 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:31.831970 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.831976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.831980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.832595 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.833192 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.833199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.833206 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.833417 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.834094 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:31.834102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:31.834389 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 14:53:31.834597 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 14:53:31.835106 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":545,"threads_started":1}
I20260620 14:53:31.841555 23481 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:36817
I20260620 14:53:31.841590 23616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:36817 every 8 connection(s)
I20260620 14:53:31.842919 23481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:31.843775 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 23481
I20260620 14:53:31.844056 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:31.844682 22143 external_mini_cluster.cc:1428] Setting key 0f18c6474b703de8d67566aa68368498
I20260620 14:53:31.845564 23617 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36537
I20260620 14:53:31.845831 23617 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:31.846575 23617 heartbeater.cc:511] Master 127.21.159.254:36537 requested a full tablet report, sending...
I20260620 14:53:31.848433 23270 ts_manager.cc:194] Registered new tserver with Master: bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194:36817)
I20260620 14:53:31.849411 23270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:51729
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:31.888477 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36537
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:32.245013 23621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:32.245337 23621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:32.245535 23621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:32.253865 23621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:32.254012 23621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:32.254194 23621 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:53:32.254292 23621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:32.264983 23621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:36537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:32.266376 23621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:32.267903 23621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:32.282387 23627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:32.282500 23629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:32.282649 23626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:32.283303 23621 server_base.cc:1034] running on GCE node
I20260620 14:53:32.283982 23621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:32.285111 23621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:32.286368 23621 hybrid_clock.cc:648] HybridClock initialized: now 1781967212286274 us; error 64 us; skew 500 ppm
I20260620 14:53:32.286763 23621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:53:32.287402 23477 heartbeater.cc:503] Master 127.21.159.254:36537 was elected leader, sending a full tablet report...
Jun 20 14:53:32 dist-test-slave-3mqx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967212, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:32.290845 23621 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:32.292689 23621 webserver.cc:460] Webserver started at http://127.21.159.195:36523/ using document root <none> and password file <none>
I20260620 14:53:32.293457 23621 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:32.293584 23621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:32.293892 23621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:32.297155 23621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "b3ffff01194443019f3a9262fe9ce3c5"
format_stamp: "Formatted at 2026-06-20 14:53:32 on dist-test-slave-3mqx"
server_key: "6eacfa68105939b7e915d24d272b548e"
server_key_iv: "ab32a9946104492f4cb7ff9e96b6bb8f"
server_key_version: "encryptionkey@0"
I20260620 14:53:32.298074 23621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "b3ffff01194443019f3a9262fe9ce3c5"
format_stamp: "Formatted at 2026-06-20 14:53:32 on dist-test-slave-3mqx"
server_key: "6eacfa68105939b7e915d24d272b548e"
server_key_iv: "ab32a9946104492f4cb7ff9e96b6bb8f"
server_key_version: "encryptionkey@0"
I20260620 14:53:32.304780 23621 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 14:53:32.309161 23636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:32.310869 23621 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:53:32.311182 23621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "b3ffff01194443019f3a9262fe9ce3c5"
format_stamp: "Formatted at 2026-06-20 14:53:32 on dist-test-slave-3mqx"
server_key: "6eacfa68105939b7e915d24d272b548e"
server_key_iv: "ab32a9946104492f4cb7ff9e96b6bb8f"
server_key_version: "encryptionkey@0"
I20260620 14:53:32.311443 23621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:32.326829 23621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:32.331737 23621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:32.332039 23621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:32.334813 23621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:32.334954 23621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:32.335198 23621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:32.335289 23621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:53:32 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967212, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:32.366914 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.336718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36537 (local address 127.21.159.195:58867)
0620 14:53:32.337686 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.337746 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.340125 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.341448 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.341463 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.342444 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:32.343574 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.343613 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.345250 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.345256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.347519 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.347535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.348972 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.348984 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.349429 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.350440 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:32.350483 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:32.354797 (+ 4314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:32.360180 (+ 5383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:32.360198 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:32.360221 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:32.361076 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:32.361686 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:32.361694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:32.361697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:32.362032 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:32.362865 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:32.362880 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:32.363538 (+ 658us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.363934 (+ 396us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.364679 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":391,"threads_started":1}
I20260620 14:53:32.367898 23645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.336079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:58867 (local address 127.21.159.254:36537)
0620 14:53:32.336719 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.336724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.337884 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 14:53:32.340542 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.340548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.340653 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.340749 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:32.343852 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.345048 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.347716 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.348640 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.348806 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.355286 (+ 6480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:32.355296 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:32.355305 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:32.355365 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:32.359935 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.361274 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.361279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.361281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.361505 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.362210 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.362217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.362222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.362698 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:32.362865 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.367043 (+ 4178us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.367381 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":364,"threads_started":1}
I20260620 14:53:32.385476 23621 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:32839
I20260620 14:53:32.385492 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:32839 every 8 connection(s)
I20260620 14:53:32.387261 23621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:32.389895 23757 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36537
I20260620 14:53:32.390182 23757 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:32.391162 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 23621
I20260620 14:53:32.391206 23757 heartbeater.cc:511] Master 127.21.159.254:36537 requested a full tablet report, sending...
I20260620 14:53:32.391367 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:32.392171 22143 external_mini_cluster.cc:1428] Setting key 4486d0423a73139dc33ff8670d017ea4
I20260620 14:53:32.393479 23270 ts_manager.cc:194] Registered new tserver with Master: b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839)
I20260620 14:53:32.394775 23270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:58867
I20260620 14:53:32.396662 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:32.400228 23645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.375377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:50733 (local address 127.21.159.254:36537)
0620 14:53:32.375589 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.375597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.380191 (+ 4594us) server_negotiation.cc:407] Connection header received
0620 14:53:32.380515 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.380523 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.380636 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.380788 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:32.382617 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.383832 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.385983 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.386639 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.386805 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.389441 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:32.389450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:32.389455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:32.389494 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:32.396954 (+ 7460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.397937 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.397943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.397947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.398109 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.398537 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.398542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.398546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.398922 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:32.399093 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.399610 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.399844 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:53:32.400386 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.374984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36537 (local address 127.21.159.195:50733)
0620 14:53:32.379957 (+ 4973us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.380008 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.380276 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.381045 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.381057 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.381837 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:32.382403 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.382421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.384012 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.384021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.385825 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.385839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.386860 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.386864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.387004 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.387861 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:32.387896 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:32.389238 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:32.397146 (+ 7908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:32.397155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:32.397159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:32.397774 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:32.398242 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:32.398247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:32.398251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:32.398407 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:32.399163 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:32.399168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:32.399362 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.399522 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.399940 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4704,"thread_start_us":4337,"threads_started":1}
Jun 20 14:53:32 dist-test-slave-3mqx krb5kdc[23225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:53:32 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:32.446549 23645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.431265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.21.159.254:36537)
0620 14:53:32.431508 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.431515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.431840 (+ 325us) server_negotiation.cc:407] Connection header received
0620 14:53:32.432039 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.432043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.432123 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.432227 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:32.433699 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.434420 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.435880 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.436311 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.436413 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.440824 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:32.440830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:32.440833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:32.440874 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:32.443662 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.444432 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.444442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.444444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.444580 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.445024 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.445029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.445031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.445331 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:32.445641 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.445961 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.446197 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:53:32.451228 23269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:32.458455 23269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:32.497660 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.472465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.21.159.194:36817)
0620 14:53:32.488774 (+ 16309us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.488783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.488819 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:53:32.488917 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.488927 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.489163 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.489372 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.490365 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.491808 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.494873 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.495992 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.496213 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.497075 (+ 862us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.497150 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.497355 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16148,"thread_start_us":16042,"threads_started":1}
I20260620 14:53:32.498355 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.471946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36817 (local address 127.0.0.1:54532)
0620 14:53:32.488588 (+ 16642us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.488610 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.488750 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.489595 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.489602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.489632 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.490050 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.490132 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.491949 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.491954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.494671 (+ 2717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.494685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.496214 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.496221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.496409 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.496469 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.496605 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.496835 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16524,"mutex_wait_us":178,"thread_start_us":16385,"threads_started":1}
I20260620 14:53:32.498554 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.471883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:38161 (local address 127.0.0.1:42454)
0620 14:53:32.487760 (+ 15877us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.487843 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.488133 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.489033 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.489042 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.489080 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.489602 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.489614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.491670 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.491678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.494664 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.494685 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.497361 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.497367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.497627 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.497731 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.497959 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.498117 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15474,"mutex_wait_us":232,"thread_start_us":17279,"threads_started":1}
I20260620 14:53:32.498683 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.472816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.21.159.195:32839)
0620 14:53:32.486890 (+ 14074us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.486903 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.489604 (+ 2701us) server_negotiation.cc:407] Connection header received
0620 14:53:32.489716 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.489726 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.489962 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.490151 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.491082 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.492474 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.495255 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.496814 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.497013 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.497172 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.498148 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.498360 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13846,"thread_start_us":13622,"threads_started":1}
I20260620 14:53:32.499186 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.472333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:32839 (local address 127.0.0.1:47032)
0620 14:53:32.489300 (+ 16967us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.489324 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.489446 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.490470 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.490475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.490500 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.490848 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.490859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.492655 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.492660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.495049 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.495064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.497336 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.497348 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.497624 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.497725 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.498073 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.498205 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16781,"mutex_wait_us":106,"thread_start_us":14829,"threads_started":1}
I20260620 14:53:32.499668 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.471958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.21.159.193:38161)
0620 14:53:32.486278 (+ 14320us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.486291 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.488005 (+ 1714us) server_negotiation.cc:407] Connection header received
0620 14:53:32.488438 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.488451 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.488762 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.488993 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.489817 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.491543 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.495202 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.496814 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.496991 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.497160 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.499325 (+ 2165us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.499466 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14147,"thread_start_us":13990,"threads_started":1}
I20260620 14:53:32.502988 23547 tablet_service.cc:1467] Processing CreateTablet for tablet 4bec35022dcb4bb080a79ff021b4a33e (DEFAULT_TABLE table=test-table [id=d717b5f13a60467fb48072f1d0a87644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.503010 23687 tablet_service.cc:1467] Processing CreateTablet for tablet 4bec35022dcb4bb080a79ff021b4a33e (DEFAULT_TABLE table=test-table [id=d717b5f13a60467fb48072f1d0a87644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.504415 23547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bec35022dcb4bb080a79ff021b4a33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.504508 23687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bec35022dcb4bb080a79ff021b4a33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.506724 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 4bec35022dcb4bb080a79ff021b4a33e (DEFAULT_TABLE table=test-table [id=d717b5f13a60467fb48072f1d0a87644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.508291 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bec35022dcb4bb080a79ff021b4a33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.518101 23776 tablet_bootstrap.cc:492] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Bootstrap starting.
I20260620 14:53:32.520745 23777 tablet_bootstrap.cc:492] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Bootstrap starting.
I20260620 14:53:32.524187 23778 tablet_bootstrap.cc:492] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Bootstrap starting.
I20260620 14:53:32.524602 23776 tablet_bootstrap.cc:654] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.525067 23777 tablet_bootstrap.cc:654] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.526510 23777 log.cc:824] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:32.526512 23776 log.cc:824] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:32.527947 23778 tablet_bootstrap.cc:654] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.529299 23778 log.cc:824] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:32.531245 23776 tablet_bootstrap.cc:492] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: No bootstrap required, opened a new log
I20260620 14:53:32.531688 23776 ts_tablet_manager.cc:1397] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260620 14:53:32.533545 23778 tablet_bootstrap.cc:492] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: No bootstrap required, opened a new log
I20260620 14:53:32.533882 23778 ts_tablet_manager.cc:1397] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 14:53:32.536454 23777 tablet_bootstrap.cc:492] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: No bootstrap required, opened a new log
I20260620 14:53:32.536798 23777 ts_tablet_manager.cc:1397] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260620 14:53:32.544250 23778 raft_consensus.cc:348] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.544746 23778 raft_consensus.cc:374] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.544847 23778 raft_consensus.cc:729] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d9b84130d6417c83f0f707b0628a9d, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.545562 23778 consensus_queue.cc:260] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.546062 23776 raft_consensus.cc:348] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.546592 23776 raft_consensus.cc:374] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.546705 23776 raft_consensus.cc:729] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc583c84da1641c0b85aa6ccf16a443a, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.547482 23776 consensus_queue.cc:260] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.548666 23617 heartbeater.cc:503] Master 127.21.159.254:36537 was elected leader, sending a full tablet report...
I20260620 14:53:32.551131 23776 ts_tablet_manager.cc:1428] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Time spent starting tablet: real 0.019s user 0.014s sys 0.003s
I20260620 14:53:32.550880 23777 raft_consensus.cc:348] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.551455 23777 raft_consensus.cc:374] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.551585 23777 raft_consensus.cc:729] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ffff01194443019f3a9262fe9ce3c5, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.552392 23777 consensus_queue.cc:260] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.554050 23757 heartbeater.cc:503] Master 127.21.159.254:36537 was elected leader, sending a full tablet report...
I20260620 14:53:32.554752 23778 ts_tablet_manager.cc:1428] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Time spent starting tablet: real 0.019s user 0.014s sys 0.000s
I20260620 14:53:32.554804 23777 ts_tablet_manager.cc:1428] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
W20260620 14:53:32.595498 23618 tablet.cc:2369] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:32.618177 23784 raft_consensus.cc:482] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:32.618510 23784 raft_consensus.cc:504] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.620353 23784 leader_election.cc:288] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161), bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194:36817)
I20260620 14:53:32.626873 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.620930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:35985 (local address 127.21.159.194:36817)
0620 14:53:32.621148 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.621154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.621177 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:53:32.621352 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.621358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.621444 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.621558 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.622112 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.623104 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.624896 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.625924 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.626049 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.626101 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.626529 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.626619 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:53:32.627076 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.620634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36817 (local address 127.21.159.195:35985)
0620 14:53:32.620984 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.621006 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.621171 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.621636 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.621640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.621666 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.621952 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.621963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.623237 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.623244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.624739 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.624750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.626099 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.626105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.626348 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.626392 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.626512 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.626741 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260620 14:53:32.628089 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.620829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:46357 (local address 127.21.159.193:38161)
0620 14:53:32.621044 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.621050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.621765 (+ 715us) server_negotiation.cc:407] Connection header received
0620 14:53:32.621907 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.621914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.621986 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.622075 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.622693 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.623716 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.625635 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.626914 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.627132 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.627216 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.627725 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.627858 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:53:32.628191 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.620846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:38161 (local address 127.21.159.195:46357)
0620 14:53:32.621628 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.621646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.621774 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.622103 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.622107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.622144 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.622484 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.622495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.623853 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.623857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.625436 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.625449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.627360 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.627369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.627567 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.627591 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.627697 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.627847 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":434,"threads_started":1}
I20260620 14:53:32.628422 23567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec35022dcb4bb080a79ff021b4a33e" candidate_uuid: "b3ffff01194443019f3a9262fe9ce3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc583c84da1641c0b85aa6ccf16a443a" is_pre_election: true
I20260620 14:53:32.629128 23567 raft_consensus.cc:2454] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ffff01194443019f3a9262fe9ce3c5 in term 0.
I20260620 14:53:32.629365 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec35022dcb4bb080a79ff021b4a33e" candidate_uuid: "b3ffff01194443019f3a9262fe9ce3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d9b84130d6417c83f0f707b0628a9d" is_pre_election: true
I20260620 14:53:32.629969 23431 raft_consensus.cc:2454] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3ffff01194443019f3a9262fe9ce3c5 in term 0.
I20260620 14:53:32.629997 23640 leader_election.cc:302] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3ffff01194443019f3a9262fe9ce3c5, bc583c84da1641c0b85aa6ccf16a443a; no voters:
I20260620 14:53:32.630496 23784 raft_consensus.cc:2781] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:32.630676 23784 raft_consensus.cc:482] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:32.630810 23784 raft_consensus.cc:3037] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:32.633710 23784 raft_consensus.cc:504] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.634420 23784 leader_election.cc:288] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [CANDIDATE]: Term 1 election: Requested vote from peers 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161), bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194:36817)
I20260620 14:53:32.634956 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec35022dcb4bb080a79ff021b4a33e" candidate_uuid: "b3ffff01194443019f3a9262fe9ce3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d9b84130d6417c83f0f707b0628a9d"
I20260620 14:53:32.635113 23567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec35022dcb4bb080a79ff021b4a33e" candidate_uuid: "b3ffff01194443019f3a9262fe9ce3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc583c84da1641c0b85aa6ccf16a443a"
I20260620 14:53:32.635249 23431 raft_consensus.cc:3037] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:32.635389 23567 raft_consensus.cc:3037] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:32.637791 23567 raft_consensus.cc:2454] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ffff01194443019f3a9262fe9ce3c5 in term 1.
I20260620 14:53:32.637771 23431 raft_consensus.cc:2454] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3ffff01194443019f3a9262fe9ce3c5 in term 1.
I20260620 14:53:32.638301 23640 leader_election.cc:302] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64d9b84130d6417c83f0f707b0628a9d, b3ffff01194443019f3a9262fe9ce3c5, bc583c84da1641c0b85aa6ccf16a443a; no voters:
I20260620 14:53:32.638734 23784 raft_consensus.cc:2781] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:32.639344 23784 raft_consensus.cc:686] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 1 LEADER]: Becoming Leader. State: Replica: b3ffff01194443019f3a9262fe9ce3c5, State: Running, Role: LEADER
W20260620 14:53:32.639375 23758 tablet.cc:2369] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:32.639982 23784 consensus_queue.cc:237] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.645414 23268 catalog_manager.cc:5495] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195). New cstate: current_term: 1 leader_uuid: "b3ffff01194443019f3a9262fe9ce3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } health_report { overall_health: HEALTHY } } }
I20260620 14:53:32.706187 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.699857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.21.159.195:32839)
0620 14:53:32.700097 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.700104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.700129 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:53:32.700464 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.700468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.700559 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.700732 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:32.701197 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.701959 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.703534 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.704071 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.704200 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.704748 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:32.705404 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:32.705546 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.705752 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.705876 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:53:32.715862 23431 raft_consensus.cc:1261] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Refusing update from remote peer b3ffff01194443019f3a9262fe9ce3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:32.715866 23567 raft_consensus.cc:1261] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Refusing update from remote peer b3ffff01194443019f3a9262fe9ce3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:32.717039 23786 consensus_queue.cc:1035] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:32.717458 23784 consensus_queue.cc:1035] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:32.732031 23791 mvcc.cc:205] Tried to move back new op lower bound from 7298937703271514112 to 7298937702977916928. Current Snapshot: MvccSnapshot[applied={T|T < 7298937703271514112}]
I20260620 14:53:32.732064 23790 mvcc.cc:205] Tried to move back new op lower bound from 7298937703271514112 to 7298937702977916928. Current Snapshot: MvccSnapshot[applied={T|T < 7298937703271514112}]
I20260620 14:53:32.752326 23268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:32.752729 23268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:32.756817 23268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf: Sending DeleteTablet for 3 replicas of tablet 4bec35022dcb4bb080a79ff021b4a33e
I20260620 14:53:32.758195 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bec35022dcb4bb080a79ff021b4a33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:32 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:42454
I20260620 14:53:32.758195 23547 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bec35022dcb4bb080a79ff021b4a33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:32 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54532
I20260620 14:53:32.759209 23687 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bec35022dcb4bb080a79ff021b4a33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:32 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:47032
I20260620 14:53:32.760788 23268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:32.761547 23268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:32.764340 23800 tablet_replica.cc:330] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: stopping tablet replica
I20260620 14:53:32.764879 23800 raft_consensus.cc:2229] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:32.765453 23800 raft_consensus.cc:2258] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:32.766310 23799 tablet_replica.cc:330] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: stopping tablet replica
I20260620 14:53:32.766765 23799 raft_consensus.cc:2229] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:32.767308 23799 raft_consensus.cc:2258] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:32.768354 23800 ts_tablet_manager.cc:1905] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:32.769338 23799 ts_tablet_manager.cc:1905] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:32.771023 23686 tablet_service.cc:1467] Processing CreateTablet for tablet af1f686228a141c29092703d23be8e10 (DEFAULT_TABLE table=test-table [id=05275e9cfd284b508ce0ce3f37cce8c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.771159 23547 tablet_service.cc:1467] Processing CreateTablet for tablet af1f686228a141c29092703d23be8e10 (DEFAULT_TABLE table=test-table [id=05275e9cfd284b508ce0ce3f37cce8c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.772195 23686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1f686228a141c29092703d23be8e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.775488 23407 tablet_service.cc:1467] Processing CreateTablet for tablet af1f686228a141c29092703d23be8e10 (DEFAULT_TABLE table=test-table [id=05275e9cfd284b508ce0ce3f37cce8c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.776252 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1f686228a141c29092703d23be8e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.781831 23801 tablet_replica.cc:330] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: stopping tablet replica
I20260620 14:53:32.782389 23801 raft_consensus.cc:2229] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:32.783300 23801 raft_consensus.cc:2258] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:32.784597 23799 ts_tablet_manager.cc:1918] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:32.786592 23799 log.cc:1196] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/4bec35022dcb4bb080a79ff021b4a33e
I20260620 14:53:32.787357 23799 ts_tablet_manager.cc:1939] T 4bec35022dcb4bb080a79ff021b4a33e P 64d9b84130d6417c83f0f707b0628a9d: Deleting consensus metadata
I20260620 14:53:32.788589 23547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1f686228a141c29092703d23be8e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.789078 23801 ts_tablet_manager.cc:1905] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:32.789343 23255 catalog_manager.cc:4841] TS 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161): tablet 4bec35022dcb4bb080a79ff021b4a33e (table test-table [id=d717b5f13a60467fb48072f1d0a87644]) successfully deleted
I20260620 14:53:32.791339 23777 tablet_bootstrap.cc:492] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: Bootstrap starting.
I20260620 14:53:32.793124 23777 tablet_bootstrap.cc:654] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.795562 23776 tablet_bootstrap.cc:492] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: Bootstrap starting.
I20260620 14:53:32.795637 23777 tablet_bootstrap.cc:492] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: No bootstrap required, opened a new log
I20260620 14:53:32.795779 23777 ts_tablet_manager.cc:1397] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:53:32.796608 23777 raft_consensus.cc:348] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.796839 23777 raft_consensus.cc:374] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.796926 23777 raft_consensus.cc:729] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ffff01194443019f3a9262fe9ce3c5, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.797199 23777 consensus_queue.cc:260] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.797840 23776 tablet_bootstrap.cc:654] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.797871 23777 ts_tablet_manager.cc:1428] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:32.801559 23776 tablet_bootstrap.cc:492] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: No bootstrap required, opened a new log
I20260620 14:53:32.801817 23776 ts_tablet_manager.cc:1397] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 14:53:32.802205 23800 ts_tablet_manager.cc:1918] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:32.802368 23800 log.cc:1196] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/4bec35022dcb4bb080a79ff021b4a33e
I20260620 14:53:32.802945 23800 ts_tablet_manager.cc:1939] T 4bec35022dcb4bb080a79ff021b4a33e P bc583c84da1641c0b85aa6ccf16a443a: Deleting consensus metadata
I20260620 14:53:32.804286 23776 raft_consensus.cc:348] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.804621 23776 raft_consensus.cc:374] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.804770 23776 raft_consensus.cc:729] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc583c84da1641c0b85aa6ccf16a443a, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.805007 23257 catalog_manager.cc:4841] TS bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194:36817): tablet 4bec35022dcb4bb080a79ff021b4a33e (table test-table [id=d717b5f13a60467fb48072f1d0a87644]) successfully deleted
I20260620 14:53:32.805351 23778 tablet_bootstrap.cc:492] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Bootstrap starting.
I20260620 14:53:32.805683 23776 consensus_queue.cc:260] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.807130 23778 tablet_bootstrap.cc:654] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.807874 23801 ts_tablet_manager.cc:1918] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:32.808059 23801 log.cc:1196] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/4bec35022dcb4bb080a79ff021b4a33e
I20260620 14:53:32.808938 23801 ts_tablet_manager.cc:1939] T 4bec35022dcb4bb080a79ff021b4a33e P b3ffff01194443019f3a9262fe9ce3c5: Deleting consensus metadata
I20260620 14:53:32.808998 23776 ts_tablet_manager.cc:1428] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:53:32.812569 23254 catalog_manager.cc:4841] TS b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839): tablet 4bec35022dcb4bb080a79ff021b4a33e (table test-table [id=d717b5f13a60467fb48072f1d0a87644]) successfully deleted
I20260620 14:53:32.814306 23778 tablet_bootstrap.cc:492] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: No bootstrap required, opened a new log
I20260620 14:53:32.814455 23778 ts_tablet_manager.cc:1397] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 14:53:32.815327 23778 raft_consensus.cc:348] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.815598 23778 raft_consensus.cc:374] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.815704 23778 raft_consensus.cc:729] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d9b84130d6417c83f0f707b0628a9d, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.815996 23778 consensus_queue.cc:260] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:32.816538 23778 ts_tablet_manager.cc:1428] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260620 14:53:33.029060 23478 tablet.cc:2369] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:33.055676 23783 raft_consensus.cc:482] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:33.055981 23783 raft_consensus.cc:504] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.058327 23783 leader_election.cc:288] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161), b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839)
I20260620 14:53:33.068601 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.058677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:53597 (local address 127.21.159.193:38161)
0620 14:53:33.058986 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.058996 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.060261 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 14:53:33.060385 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.060392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.060477 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.060581 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:33.061469 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.062467 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.066006 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.067420 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.067640 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.067729 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.068124 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.068276 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 14:53:33.068665 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.059246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:38161 (local address 127.21.159.194:53597)
0620 14:53:33.060067 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:33.060082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:33.060257 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:33.060728 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:33.060749 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:33.060769 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:33.061303 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.061316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.062600 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.062606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.065762 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.065783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.067717 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.067725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.067917 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.067954 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:53:33.068066 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:53:33.068226 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":145,"thread_start_us":496,"threads_started":1}
I20260620 14:53:33.069459 23782 raft_consensus.cc:482] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:33.069672 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1f686228a141c29092703d23be8e10" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d9b84130d6417c83f0f707b0628a9d" is_pre_election: true
I20260620 14:53:33.069850 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.058833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:41851 (local address 127.21.159.195:32839)
0620 14:53:33.059046 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.059092 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.059914 (+ 822us) server_negotiation.cc:407] Connection header received
0620 14:53:33.060257 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.060264 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.060403 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.060510 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:33.061333 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.062800 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.067224 (+ 4424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.068492 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.068663 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.068750 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.069335 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.069498 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:53:33.069746 23782 raft_consensus.cc:504] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.070560 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.058639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:32839 (local address 127.21.159.194:41851)
0620 14:53:33.059750 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:33.059775 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:33.060060 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:33.060525 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:33.060530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:33.060562 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:33.061088 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.061102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.063405 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.063414 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.065762 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.065783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.068916 (+ 3133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.068925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.069131 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.069161 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:53:33.070031 (+ 870us) client_negotiation.cc:240] Negotiation successful
0620 14:53:33.070184 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":86,"thread_start_us":723,"threads_started":1}
I20260620 14:53:33.070575 23431 raft_consensus.cc:2454] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 0.
I20260620 14:53:33.071621 23498 leader_election.cc:302] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d9b84130d6417c83f0f707b0628a9d, bc583c84da1641c0b85aa6ccf16a443a; no voters:
I20260620 14:53:33.071652 23707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1f686228a141c29092703d23be8e10" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ffff01194443019f3a9262fe9ce3c5" is_pre_election: true
I20260620 14:53:33.072100 23783 raft_consensus.cc:2781] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:33.072216 23783 raft_consensus.cc:482] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:33.072137 23707 raft_consensus.cc:2454] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 0.
I20260620 14:53:33.072239 23782 leader_election.cc:288] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194:36817), b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839)
I20260620 14:53:33.072304 23783 raft_consensus.cc:3037] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:33.078104 23783 raft_consensus.cc:504] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.080045 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1f686228a141c29092703d23be8e10" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d9b84130d6417c83f0f707b0628a9d"
I20260620 14:53:33.080343 23431 raft_consensus.cc:3037] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:33.083943 23431 raft_consensus.cc:2454] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 1.
I20260620 14:53:33.084511 23498 leader_election.cc:302] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d9b84130d6417c83f0f707b0628a9d, bc583c84da1641c0b85aa6ccf16a443a; no voters:
I20260620 14:53:33.086576 23783 leader_election.cc:288] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 election: Requested vote from peers 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161), b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839)
I20260620 14:53:33.086531 23707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1f686228a141c29092703d23be8e10" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ffff01194443019f3a9262fe9ce3c5"
I20260620 14:53:33.086686 23809 raft_consensus.cc:2781] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:33.086796 23707 raft_consensus.cc:3037] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:33.086923 23809 raft_consensus.cc:686] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 1 LEADER]: Becoming Leader. State: Replica: bc583c84da1641c0b85aa6ccf16a443a, State: Running, Role: LEADER
I20260620 14:53:33.087460 23809 consensus_queue.cc:237] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.091171 23707 raft_consensus.cc:2454] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 1.
I20260620 14:53:33.093438 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.073268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:47615 (local address 127.21.159.194:36817)
0620 14:53:33.073513 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.073522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.075026 (+ 1504us) server_negotiation.cc:407] Connection header received
0620 14:53:33.075739 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.075745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.075831 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.075898 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:33.076937 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.079140 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.083793 (+ 4653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.085607 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.085831 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.085948 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.092695 (+ 6747us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.092933 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:53:33.094789 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.072693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36817 (local address 127.21.159.193:47615)
0620 14:53:33.074924 (+ 2231us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:33.074944 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:33.075565 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:33.076131 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:33.076139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:33.076181 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:33.076682 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.076698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.080056 (+ 3358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.080065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.083469 (+ 3404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.083497 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.092059 (+ 8562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.092075 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.092392 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.092456 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:53:33.093992 (+ 1536us) client_negotiation.cc:240] Negotiation successful
0620 14:53:33.094232 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2093,"thread_start_us":1930,"threads_started":1}
I20260620 14:53:33.096053 23567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1f686228a141c29092703d23be8e10" candidate_uuid: "64d9b84130d6417c83f0f707b0628a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc583c84da1641c0b85aa6ccf16a443a" is_pre_election: true
I20260620 14:53:33.096380 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.075295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:54639 (local address 127.21.159.195:32839)
0620 14:53:33.075829 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.075835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.087788 (+ 11953us) server_negotiation.cc:407] Connection header received
0620 14:53:33.088045 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.088052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.088141 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.088210 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:33.089094 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.090310 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.093371 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.094759 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.094927 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.095021 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.095882 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.096064 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260620 14:53:33.097446 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.075273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:32839 (local address 127.21.159.193:54639)
0620 14:53:33.087660 (+ 12387us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:33.087685 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:33.087905 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:33.088456 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:33.088462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:33.088494 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:33.088879 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.088893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.090571 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.090577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.093168 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.093185 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.095276 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.095286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.095649 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.095691 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:53:33.096625 (+ 934us) client_negotiation.cc:240] Negotiation successful
0620 14:53:33.097053 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12226,"thread_start_us":13648,"threads_started":1}
I20260620 14:53:33.098701 23707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1f686228a141c29092703d23be8e10" candidate_uuid: "64d9b84130d6417c83f0f707b0628a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ffff01194443019f3a9262fe9ce3c5" is_pre_election: true
I20260620 14:53:33.099108 23707 raft_consensus.cc:2379] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 64d9b84130d6417c83f0f707b0628a9d in current term 1: Already voted for candidate bc583c84da1641c0b85aa6ccf16a443a in this term.
I20260620 14:53:33.099982 23357 leader_election.cc:302] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64d9b84130d6417c83f0f707b0628a9d; no voters: b3ffff01194443019f3a9262fe9ce3c5, bc583c84da1641c0b85aa6ccf16a443a
I20260620 14:53:33.100724 23782 raft_consensus.cc:2726] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:53:33.101805 23270 catalog_manager.cc:5495] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a reported cstate change: term changed from 0 to 1, leader changed from <none> to bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194). New cstate: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:53:33 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:33.195147 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.177730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36537 (local address 127.0.0.1:54794)
0620 14:53:33.178662 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:33.178680 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:33.178819 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:33.179211 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:33.179214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:33.179950 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:33.180592 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.180605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.182188 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.182192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.183308 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.183322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.184380 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.184384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.184539 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.185299 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:33.185316 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:33.187631 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:33.192494 (+ 4863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:33.192507 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:33.192514 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:33.192999 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:33.193419 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:33.193422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:33.193425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:33.193566 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:33.194234 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:33.194241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:33.194395 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 14:53:33.194547 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:53:33.194768 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":630,"threads_started":1}
I20260620 14:53:33.195530 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.177740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54794 (local address 127.21.159.254:36537)
0620 14:53:33.178424 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.178433 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.178782 (+ 349us) server_negotiation.cc:407] Connection header received
0620 14:53:33.178942 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.178947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.179023 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.179185 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:33.180799 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.182024 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.183466 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.184186 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.184323 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.187865 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:33.187872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:33.187877 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:33.187915 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:33.192276 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.193135 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.193141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.193145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.193301 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.193680 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.193686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.193689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.194092 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:33.194235 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.194738 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.194989 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":343,"threads_started":1}
I20260620 14:53:33.201148 23270 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54794:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:53:33.202275 23270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:33.210620 23407 tablet_service.cc:1467] Processing CreateTablet for tablet 74b9b34a90bb45fa8d6da4602c02204e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98778a6c4bb349c8bcf53822a4df31a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:53:33.210727 23547 tablet_service.cc:1467] Processing CreateTablet for tablet 74b9b34a90bb45fa8d6da4602c02204e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98778a6c4bb349c8bcf53822a4df31a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:53:33.211309 23547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b9b34a90bb45fa8d6da4602c02204e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:33.211309 23407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b9b34a90bb45fa8d6da4602c02204e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:33.215402 23686 tablet_service.cc:1467] Processing CreateTablet for tablet 74b9b34a90bb45fa8d6da4602c02204e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98778a6c4bb349c8bcf53822a4df31a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:53:33.216037 23686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b9b34a90bb45fa8d6da4602c02204e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:33.219564 23778 tablet_bootstrap.cc:492] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: Bootstrap starting.
I20260620 14:53:33.220999 23778 tablet_bootstrap.cc:654] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:33.221642 23776 tablet_bootstrap.cc:492] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: Bootstrap starting.
I20260620 14:53:33.223969 23776 tablet_bootstrap.cc:654] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:33.224537 23777 tablet_bootstrap.cc:492] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: Bootstrap starting.
I20260620 14:53:33.226235 23777 tablet_bootstrap.cc:654] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:33.233881 23776 tablet_bootstrap.cc:492] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: No bootstrap required, opened a new log
I20260620 14:53:33.234002 23777 tablet_bootstrap.cc:492] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: No bootstrap required, opened a new log
I20260620 14:53:33.234079 23778 tablet_bootstrap.cc:492] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: No bootstrap required, opened a new log
I20260620 14:53:33.234119 23776 ts_tablet_manager.cc:1397] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 14:53:33.234194 23777 ts_tablet_manager.cc:1397] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 14:53:33.234246 23778 ts_tablet_manager.cc:1397] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260620 14:53:33.234921 23776 raft_consensus.cc:348] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.235016 23777 raft_consensus.cc:348] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.235015 23778 raft_consensus.cc:348] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.235224 23776 raft_consensus.cc:374] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:33.235299 23776 raft_consensus.cc:729] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc583c84da1641c0b85aa6ccf16a443a, State: Initialized, Role: FOLLOWER
I20260620 14:53:33.235298 23777 raft_consensus.cc:374] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:33.235296 23778 raft_consensus.cc:374] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:33.235376 23778 raft_consensus.cc:729] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d9b84130d6417c83f0f707b0628a9d, State: Initialized, Role: FOLLOWER
I20260620 14:53:33.235387 23777 raft_consensus.cc:729] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ffff01194443019f3a9262fe9ce3c5, State: Initialized, Role: FOLLOWER
I20260620 14:53:33.235602 23776 consensus_queue.cc:260] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.235657 23777 consensus_queue.cc:260] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.236263 23776 ts_tablet_manager.cc:1428] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:33.236271 23777 ts_tablet_manager.cc:1428] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:33.236121 23809 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.236445 23809 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:33.236568 23784 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.236819 23784 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:33.237068 23778 consensus_queue.cc:260] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.237689 23778 ts_tablet_manager.cc:1428] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:53:33.237757 23782 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.238058 23782 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:33.358229 23809 raft_consensus.cc:482] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:33.358484 23809 raft_consensus.cc:504] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.359493 23809 leader_election.cc:288] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161), b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839)
I20260620 14:53:33.360105 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b9b34a90bb45fa8d6da4602c02204e" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d9b84130d6417c83f0f707b0628a9d" is_pre_election: true
I20260620 14:53:33.360143 23707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b9b34a90bb45fa8d6da4602c02204e" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ffff01194443019f3a9262fe9ce3c5" is_pre_election: true
I20260620 14:53:33.360358 23431 raft_consensus.cc:2454] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 0.
I20260620 14:53:33.360410 23707 raft_consensus.cc:2454] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 0.
I20260620 14:53:33.360890 23497 leader_election.cc:302] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3ffff01194443019f3a9262fe9ce3c5, bc583c84da1641c0b85aa6ccf16a443a; no voters:
I20260620 14:53:33.361222 23809 raft_consensus.cc:2781] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:33.361353 23809 raft_consensus.cc:482] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:33.361474 23809 raft_consensus.cc:3037] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:33.364189 23809 raft_consensus.cc:504] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.364830 23809 leader_election.cc:288] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 election: Requested vote from peers 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161), b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839)
I20260620 14:53:33.365413 23707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b9b34a90bb45fa8d6da4602c02204e" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3ffff01194443019f3a9262fe9ce3c5"
I20260620 14:53:33.365657 23707 raft_consensus.cc:3037] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:33.367343 23431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b9b34a90bb45fa8d6da4602c02204e" candidate_uuid: "bc583c84da1641c0b85aa6ccf16a443a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d9b84130d6417c83f0f707b0628a9d"
I20260620 14:53:33.367650 23431 raft_consensus.cc:3037] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:33.367738 23707 raft_consensus.cc:2454] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 1.
I20260620 14:53:33.368256 23497 leader_election.cc:302] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3ffff01194443019f3a9262fe9ce3c5, bc583c84da1641c0b85aa6ccf16a443a; no voters:
I20260620 14:53:33.368605 23809 raft_consensus.cc:2781] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:33.368772 23809 raft_consensus.cc:686] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [term 1 LEADER]: Becoming Leader. State: Replica: bc583c84da1641c0b85aa6ccf16a443a, State: Running, Role: LEADER
I20260620 14:53:33.369050 23809 consensus_queue.cc:237] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } }
I20260620 14:53:33.370878 23431 raft_consensus.cc:2454] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc583c84da1641c0b85aa6ccf16a443a in term 1.
I20260620 14:53:33.371179 23783 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: TxnStatusTablet state changed. Reason: New leader bc583c84da1641c0b85aa6ccf16a443a. Latest consensus state: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.371521 23783 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:53:33.372123 23822 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:53:33.372378 23822 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:53:33.372861 23268 catalog_manager.cc:5495] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a reported cstate change: term changed from 0 to 1, leader changed from <none> to bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194). New cstate: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } health_report { overall_health: UNKNOWN } } }
I20260620 14:53:33.410787 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.401664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36817 (local address 127.0.0.1:54538)
0620 14:53:33.402345 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:33.402366 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:33.402523 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:33.403344 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:33.403349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:33.403364 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:33.403542 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.403556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.405116 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.405121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.406749 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.406766 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.407964 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.407970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.408179 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.408264 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:33.410160 (+ 1896us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:33.410180 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:53:33.410262 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:53:33.410436 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490}
I20260620 14:53:33.411768 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.401926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.21.159.194:36817)
0620 14:53:33.402126 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.402132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.402707 (+ 575us) server_negotiation.cc:407] Connection header received
0620 14:53:33.402819 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.402825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.402954 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.403160 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:33.403716 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.404927 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.406933 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.407668 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.407821 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.408529 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:33.409708 (+ 1179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:33.409873 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.411356 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.411452 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:53:33.416927 23431 raft_consensus.cc:1261] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Refusing update from remote peer bc583c84da1641c0b85aa6ccf16a443a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:33.418139 23707 raft_consensus.cc:1261] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Refusing update from remote peer bc583c84da1641c0b85aa6ccf16a443a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:33.419047 23809 consensus_queue.cc:1035] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:33.421912 23809 consensus_queue.cc:1035] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:33.426322 23782 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: TxnStatusTablet state changed. Reason: New leader bc583c84da1641c0b85aa6ccf16a443a. Latest consensus state: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.426682 23782 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:33.431237 23783 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.431526 23783 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:53:33.431898 23784 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: TxnStatusTablet state changed. Reason: New leader bc583c84da1641c0b85aa6ccf16a443a. Latest consensus state: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.432263 23784 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:33.432294 23783 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.432533 23783 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P bc583c84da1641c0b85aa6ccf16a443a: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:53:33.435498 23782 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.435798 23782 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P 64d9b84130d6417c83f0f707b0628a9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:33.441821 23784 tablet_replica.cc:439] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc583c84da1641c0b85aa6ccf16a443a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 } } peers { permanent_uuid: "bc583c84da1641c0b85aa6ccf16a443a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36817 } } peers { permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 } } }
I20260620 14:53:33.442160 23784 tablet_replica.cc:442] T 74b9b34a90bb45fa8d6da4602c02204e P b3ffff01194443019f3a9262fe9ce3c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:33.461053 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.453711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.21.159.194:36817)
0620 14:53:33.453888 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.453894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.453941 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:53:33.454135 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.454140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.454257 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.454366 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:33.454862 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.456053 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.458413 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.459400 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.459566 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.460175 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:33.460342 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:33.460467 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.460636 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.460727 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:53:33.472222 23809 consensus_queue.cc:1035] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d9b84130d6417c83f0f707b0628a9d" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:33.481412 23783 consensus_queue.cc:1035] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3ffff01194443019f3a9262fe9ce3c5" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:33.484041 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.474688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:59837 (local address 127.21.159.194:36817)
0620 14:53:33.474878 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.474884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.476254 (+ 1370us) server_negotiation.cc:407] Connection header received
0620 14:53:33.476512 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.476518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.476588 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.476643 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:33.477355 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.478695 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.481441 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.482260 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.482374 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.483169 (+ 795us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:33.483319 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:33.483386 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.483582 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.483707 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:53:33.486997 23838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.475014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36817 (local address 127.21.159.194:59837)
0620 14:53:33.476152 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:33.476173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:33.476366 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:33.476803 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:33.476808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:33.476819 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:33.477194 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.477206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.478975 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.478981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.481250 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:33.481268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.482597 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.482606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:33.482817 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.482916 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:33.483457 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:33.483465 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:53:33.486298 (+ 2833us) client_negotiation.cc:240] Negotiation successful
0620 14:53:33.486491 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":335,"threads_started":1}
I20260620 14:53:33.569176 23268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:33.569496 23268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:33.573305 23268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b7b38ce986c437db1d83d8d7f4f9bcf: Sending DeleteTablet for 3 replicas of tablet af1f686228a141c29092703d23be8e10
I20260620 14:53:33.574267 23407 tablet_service.cc:1514] Processing DeleteTablet for tablet af1f686228a141c29092703d23be8e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:33 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:42454
I20260620 14:53:33.574411 23547 tablet_service.cc:1514] Processing DeleteTablet for tablet af1f686228a141c29092703d23be8e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:33 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54532
I20260620 14:53:33.574504 23686 tablet_service.cc:1514] Processing DeleteTablet for tablet af1f686228a141c29092703d23be8e10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:33 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:47032
I20260620 14:53:33.589828 23844 tablet_replica.cc:330] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: stopping tablet replica
I20260620 14:53:33.590222 23844 raft_consensus.cc:2229] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:33.591892 23845 tablet_replica.cc:330] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: stopping tablet replica
I20260620 14:53:33.592146 23845 raft_consensus.cc:2229] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:33.592458 23845 raft_consensus.cc:2258] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:33.595218 23843 tablet_replica.cc:330] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: stopping tablet replica
I20260620 14:53:33.595500 23843 raft_consensus.cc:2229] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:33.595819 23843 raft_consensus.cc:2258] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:33.602530 23845 ts_tablet_manager.cc:1905] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:33.611619 23844 raft_consensus.cc:2258] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:33.612777 23843 ts_tablet_manager.cc:1905] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 14:53:33 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:33.615965 23844 ts_tablet_manager.cc:1905] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:33.626693 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.585551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.21.159.193:38161)
0620 14:53:33.602385 (+ 16834us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.602392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.602421 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:53:33.602508 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.602517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.602604 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.602960 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:33.605456 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.606655 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.608940 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.609655 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.609795 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.616949 (+ 7154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:33.616972 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:33.616996 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:33.617054 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:33.621668 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.622883 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.622896 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.622900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.623129 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.623658 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.623664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.623669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.624194 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:33.624558 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.625919 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.626198 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16689,"thread_start_us":17221,"threads_started":1}
W20260620 14:53:33.628371 23472 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42462
I20260620 14:53:33.628679 23845 ts_tablet_manager.cc:1918] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:33.630792 23844 ts_tablet_manager.cc:1918] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:33.630997 23843 ts_tablet_manager.cc:1918] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:33.631201 23843 log.cc:1196] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/af1f686228a141c29092703d23be8e10
I20260620 14:53:33.631428 23845 log.cc:1196] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/af1f686228a141c29092703d23be8e10
I20260620 14:53:33.631765 23843 ts_tablet_manager.cc:1939] T af1f686228a141c29092703d23be8e10 P 64d9b84130d6417c83f0f707b0628a9d: Deleting consensus metadata
I20260620 14:53:33.632103 23845 ts_tablet_manager.cc:1939] T af1f686228a141c29092703d23be8e10 P b3ffff01194443019f3a9262fe9ce3c5: Deleting consensus metadata
I20260620 14:53:33.633385 23254 catalog_manager.cc:4841] TS b3ffff01194443019f3a9262fe9ce3c5 (127.21.159.195:32839): tablet af1f686228a141c29092703d23be8e10 (table test-table [id=05275e9cfd284b508ce0ce3f37cce8c1]) successfully deleted
I20260620 14:53:33.633710 23255 catalog_manager.cc:4841] TS 64d9b84130d6417c83f0f707b0628a9d (127.21.159.193:38161): tablet af1f686228a141c29092703d23be8e10 (table test-table [id=05275e9cfd284b508ce0ce3f37cce8c1]) successfully deleted
I20260620 14:53:33.633963 23844 log.cc:1196] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/af1f686228a141c29092703d23be8e10
I20260620 14:53:33.634640 23844 ts_tablet_manager.cc:1939] T af1f686228a141c29092703d23be8e10 P bc583c84da1641c0b85aa6ccf16a443a: Deleting consensus metadata
I20260620 14:53:33.636022 23257 catalog_manager.cc:4841] TS bc583c84da1641c0b85aa6ccf16a443a (127.21.159.194:36817): tablet af1f686228a141c29092703d23be8e10 (table test-table [id=05275e9cfd284b508ce0ce3f37cce8c1]) successfully deleted
I20260620 14:53:33.654520 23816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.637800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.21.159.254:36537)
0620 14:53:33.638037 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.638046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.638514 (+ 468us) server_negotiation.cc:407] Connection header received
0620 14:53:33.638757 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.638764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.638881 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.639015 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:33.640806 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.641739 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.644173 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.644824 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.645258 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.648331 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:33.648337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:33.648341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:33.648379 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:33.651560 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.652312 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.652317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.652320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.652451 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.652855 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.652864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.652866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.653201 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:33.653489 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.653852 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.654089 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:53:33.655133 23268 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54810
I20260620 14:53:33.658139 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23341
I20260620 14:53:33.675089 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.658943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.21.159.193:38161)
0620 14:53:33.659179 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.659185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.659483 (+ 298us) server_negotiation.cc:407] Connection header received
0620 14:53:33.659631 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.659638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.659726 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.659847 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:33.661066 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.662311 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.664065 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.664829 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.664976 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.667768 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:33.667776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:33.667782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:33.667834 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:33.671736 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.672610 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.672618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.672622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.672789 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.673249 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.673257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.673261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.673633 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:33.673917 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.674389 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.674666 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 14:53:33.675781 23472 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42478
W20260620 14:53:33.676425 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23341: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:33.676661 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 23341
I20260620 14:53:33.696065 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23481
Jun 20 14:53:33 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:53:33.713757 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.696903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.21.159.194:36817)
0620 14:53:33.697138 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.697145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.697258 (+ 113us) server_negotiation.cc:407] Connection header received
0620 14:53:33.697486 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.697493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.697570 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.697696 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:33.698965 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.699886 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.701272 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.701798 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.701942 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.706631 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:33.706649 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:33.706665 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:33.706728 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:33.710386 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.711341 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.711353 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.711363 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.711534 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.711963 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.711971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.711974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.712364 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:33.712735 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.713044 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.713296 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 14:53:33.714488 23612 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54552
W20260620 14:53:33.715384 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23481: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:33.715601 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 23481
I20260620 14:53:33.736893 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23621
Jun 20 14:53:33 dist-test-slave-3mqx krb5kdc[23225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967212, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:53:33.754484 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:33.737811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.21.159.195:32839)
0620 14:53:33.738295 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:33.738299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:33.738317 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:53:33.738385 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:33.738388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:33.738450 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:33.738572 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:33.740032 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.740719 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.742275 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:33.742747 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:33.742862 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:33.747927 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:33.747953 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:33.748017 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:33.748066 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:33.751262 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.752235 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.752244 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.752246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.752391 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:33.752763 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:33.752767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:33.752768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:33.753116 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:33.753342 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:33.753909 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 14:53:33.754124 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":288,"threads_started":1}
W20260620 14:53:33.755262 23752 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47050
W20260620 14:53:33.755954 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23621: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:33.756107 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 23621
I20260620 14:53:33.778622 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23238
I20260620 14:53:33.780107 23310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:33.906406 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 23238
2026-06-20T14:53:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5127 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:34 dist-test-slave-3mqx krb5kdc[23856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:34 dist-test-slave-3mqx krb5kdc[23856](info): set up 2 sockets
Jun 20 14:53:34 dist-test-slave-3mqx krb5kdc[23856](info): commencing operation
krb5kdc: starting...
W20260620 14:53:35.015523 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:35 dist-test-slave-3mqx krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:35.215343 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:39611
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:35.560271 23869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:35.560603 23869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:35.560791 23869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:35.569092 23869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:35.569247 23869 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:53:35.569331 23869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:35.569440 23869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:35.569528 23869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:35.569641 23869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:35.580556 23869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:39611
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:35.582414 23869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:35.583873 23869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:35.596776 23874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:35.596760 23877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:35.596802 23875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:35.598186 23869 server_base.cc:1034] running on GCE node
I20260620 14:53:35.599226 23869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:35.601068 23869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:35.602355 23869 hybrid_clock.cc:648] HybridClock initialized: now 1781967215602261 us; error 77 us; skew 500 ppm
I20260620 14:53:35.602758 23869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:35 dist-test-slave-3mqx krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:35.609948 23869 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:35.611954 23869 webserver.cc:460] Webserver started at http://127.21.159.254:45447/ using document root <none> and password file <none>
I20260620 14:53:35.612865 23869 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:35.613022 23869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:35.613374 23869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:35.616911 23869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "1ed0638fb874450285b0a0116ec11e25"
format_stamp: "Formatted at 2026-06-20 14:53:35 on dist-test-slave-3mqx"
server_key: "214de05ca8fbdf31916392482b1f98af"
server_key_iv: "37bde14cc6d7a76070f40abdcbf4772a"
server_key_version: "encryptionkey@0"
I20260620 14:53:35.617882 23869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "1ed0638fb874450285b0a0116ec11e25"
format_stamp: "Formatted at 2026-06-20 14:53:35 on dist-test-slave-3mqx"
server_key: "214de05ca8fbdf31916392482b1f98af"
server_key_iv: "37bde14cc6d7a76070f40abdcbf4772a"
server_key_version: "encryptionkey@0"
I20260620 14:53:35.624181 23869 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 14:53:35.628371 23884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:35.630156 23869 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:53:35.630400 23869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "1ed0638fb874450285b0a0116ec11e25"
format_stamp: "Formatted at 2026-06-20 14:53:35 on dist-test-slave-3mqx"
server_key: "214de05ca8fbdf31916392482b1f98af"
server_key_iv: "37bde14cc6d7a76070f40abdcbf4772a"
server_key_version: "encryptionkey@0"
I20260620 14:53:35.630676 23869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:35.659001 23869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:35.663797 23869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:35.664122 23869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:35.686350 23869 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:39611
I20260620 14:53:35.686372 23945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:39611 every 8 connection(s)
I20260620 14:53:35.687914 23869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:35.691998 23946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:35.692519 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 23869
I20260620 14:53:35.692721 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:35.693459 22143 external_mini_cluster.cc:1428] Setting key 0b67ca7682d1f51bbb49b8620135b285
I20260620 14:53:35.707801 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Bootstrap starting.
Jun 20 14:53:35 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:35.712342 23946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:35.713565 23946 log.cc:824] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:35.717325 23946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: No bootstrap required, opened a new log
I20260620 14:53:35.721637 23949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:35.695916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.21.159.254:39611)
0620 14:53:35.696876 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:35.696894 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:35.696944 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:53:35.698785 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:35.698818 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:35.699417 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:35.699832 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:35.701718 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:35.703913 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:35.705928 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:35.706678 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:35.706835 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:35.712189 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:35.712198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:35.712216 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:35.712264 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:35.716240 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:35.717153 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:35.717162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:35.717170 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:35.717425 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:35.717908 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:35.717913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:35.717916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:35.718656 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:35.719088 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:35.719563 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 14:53:35.720081 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":426,"threads_started":1}
I20260620 14:53:35.727303 23946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed0638fb874450285b0a0116ec11e25" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39611 } }
I20260620 14:53:35.727715 23946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:35.727797 23946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed0638fb874450285b0a0116ec11e25, State: Initialized, Role: FOLLOWER
I20260620 14:53:35.728335 23946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed0638fb874450285b0a0116ec11e25" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39611 } }
I20260620 14:53:35.728586 23946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:35.728686 23946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:35.728883 23946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:35.731488 23946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed0638fb874450285b0a0116ec11e25" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39611 } }
I20260620 14:53:35.732049 23946 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed0638fb874450285b0a0116ec11e25; no voters:
I20260620 14:53:35.732820 23946 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:35.733026 23951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:35.734122 23951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed0638fb874450285b0a0116ec11e25, State: Running, Role: LEADER
I20260620 14:53:35.734854 23951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed0638fb874450285b0a0116ec11e25" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39611 } }
I20260620 14:53:35.735435 23946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:35.737879 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed0638fb874450285b0a0116ec11e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed0638fb874450285b0a0116ec11e25" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39611 } } }
I20260620 14:53:35.738266 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:35.738979 23953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed0638fb874450285b0a0116ec11e25. Latest consensus state: current_term: 1 leader_uuid: "1ed0638fb874450285b0a0116ec11e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed0638fb874450285b0a0116ec11e25" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39611 } } }
I20260620 14:53:35.739302 23953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:35.744274 23959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:35.750555 23959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:35.761379 23959 catalog_manager.cc:1347] Generated new cluster ID: 5de031cb58334c56860980bed5cfce49
I20260620 14:53:35.761528 23959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:35.782457 23959 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:35.784708 23959 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:35.795874 23959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Generated new TSK 0
I20260620 14:53:35.797165 23959 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:35.847151 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39611
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:36.196576 23972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:36.196962 23972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:36.197134 23972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:36.205094 23972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:36.205296 23972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:36.205462 23972 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:53:36.205557 23972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:53:36.205681 23972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:36.216348 23972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:39611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:36.217793 23972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:36.219223 23972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:36.233139 23978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:36.233115 23977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:36.234462 23980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:36.234863 23972 server_base.cc:1034] running on GCE node
I20260620 14:53:36.235741 23972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:36.236984 23972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:36.238296 23972 hybrid_clock.cc:648] HybridClock initialized: now 1781967216238210 us; error 73 us; skew 500 ppm
I20260620 14:53:36.238678 23972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:36 dist-test-slave-3mqx krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967216, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:36.242187 23972 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:36.243826 23972 webserver.cc:460] Webserver started at http://127.21.159.193:39149/ using document root <none> and password file <none>
I20260620 14:53:36.244606 23972 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:36.244736 23972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:36.245070 23972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:36.248400 23972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "2f539a55a32a4894b6e4ed891ae58622"
format_stamp: "Formatted at 2026-06-20 14:53:36 on dist-test-slave-3mqx"
server_key: "946f41fe59acdb5873ac50814de9f807"
server_key_iv: "9ae47806b61bf74dae066446246734c0"
server_key_version: "encryptionkey@0"
I20260620 14:53:36.249221 23972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "2f539a55a32a4894b6e4ed891ae58622"
format_stamp: "Formatted at 2026-06-20 14:53:36 on dist-test-slave-3mqx"
server_key: "946f41fe59acdb5873ac50814de9f807"
server_key_iv: "9ae47806b61bf74dae066446246734c0"
server_key_version: "encryptionkey@0"
I20260620 14:53:36.254942 23972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:53:36.259220 23987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:36.260815 23972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:53:36.261124 23972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "2f539a55a32a4894b6e4ed891ae58622"
format_stamp: "Formatted at 2026-06-20 14:53:36 on dist-test-slave-3mqx"
server_key: "946f41fe59acdb5873ac50814de9f807"
server_key_iv: "9ae47806b61bf74dae066446246734c0"
server_key_version: "encryptionkey@0"
I20260620 14:53:36.261368 23972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:36.284830 23972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:36.289259 23972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:36.289572 23972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:36.292510 23972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:36.292656 23972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:36.292805 23972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:36.292917 23972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:53:36 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967216, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:36.319397 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.293129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:47829 (local address 127.21.159.254:39611)
0620 14:53:36.293937 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:36.293944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:36.295084 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 14:53:36.297768 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:36.297778 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:36.297884 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:36.297976 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:36.301110 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.302217 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.304725 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.305371 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.305522 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.311202 (+ 5680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:36.311208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:36.311211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:36.311250 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:36.314654 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.316112 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.316120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.316126 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.316305 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.317036 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.317040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.317043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.317324 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:36.317458 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:36.318748 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0620 14:53:36.318943 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":561,"threads_started":1}
I20260620 14:53:36.321238 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.293928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39611 (local address 127.21.159.193:47829)
0620 14:53:36.294865 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:36.294925 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:36.297365 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:36.298719 (+ 1354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:36.298728 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:36.299692 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:36.300875 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.300916 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.302397 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.302409 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.304543 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.304567 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.305656 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.305664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.306036 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.307102 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:36.307159 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:36.310867 (+ 3708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:36.314890 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.314909 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.314935 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.315899 (+ 964us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.316437 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.316445 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.316449 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.316846 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.317486 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:36.317500 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:36.318213 (+ 713us) client_negotiation.cc:769] Sending connection context
0620 14:53:36.318668 (+ 455us) client_negotiation.cc:240] Negotiation successful
0620 14:53:36.319508 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":427,"threads_started":1}
I20260620 14:53:36.338398 23972 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:36747
I20260620 14:53:36.338440 24107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:36747 every 8 connection(s)
I20260620 14:53:36.340090 23972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:36.342816 24108 heartbeater.cc:347] Connected to a master server at 127.21.159.254:39611
I20260620 14:53:36.343117 24108 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:36.343971 24108 heartbeater.cc:511] Master 127.21.159.254:39611 requested a full tablet report, sending...
I20260620 14:53:36.345899 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.330067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:32781 (local address 127.21.159.254:39611)
0620 14:53:36.330237 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:36.330244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:36.330534 (+ 290us) server_negotiation.cc:407] Connection header received
0620 14:53:36.330770 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:36.330775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:36.330856 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:36.330947 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:36.332738 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.333695 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.336202 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.336851 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.337041 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.339691 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:36.339696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:36.339700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:36.339733 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:36.342807 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.343690 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.343696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.343699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.343869 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.344336 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.344341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.344344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.344736 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:36.344903 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:36.345442 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 14:53:36.345628 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:53:36.346074 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.329544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39611 (local address 127.21.159.193:32781)
0620 14:53:36.330371 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:36.330398 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:36.330613 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:36.331025 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:36.331033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:36.331892 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:36.332517 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.332539 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.333847 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.333855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.335991 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.336021 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.337116 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.337129 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.337390 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.338209 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:36.338241 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:36.339489 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:36.342963 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.342968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.342972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.343540 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.343985 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.343988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.343990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.344170 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.344909 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:36.344914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:36.345133 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 14:53:36.345259 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:53:36.345638 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"spinlock_wait_cycles":21376,"thread_start_us":589,"threads_started":1}
I20260620 14:53:36.346729 23901 ts_manager.cc:194] Registered new tserver with Master: 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747)
I20260620 14:53:36.348310 23901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:47829
I20260620 14:53:36.349433 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 23972
I20260620 14:53:36.349651 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:36.350477 22143 external_mini_cluster.cc:1428] Setting key be456bd47386f17259867aab67c3d22d
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:36.393880 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39611
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:36.746744 24112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:36.747114 24112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:36.747290 24112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:36.755213 24112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:36.755331 24112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:36.755488 24112 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:53:36.755568 24112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:53:36.755695 24112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:36.766234 24112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:39611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:36.767717 24112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:36.769083 24112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:36.782764 24117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:36.782820 24120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:36.782770 24118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:36.783905 24112 server_base.cc:1034] running on GCE node
I20260620 14:53:36.784516 24112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:36.785593 24112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:36.786799 24112 hybrid_clock.cc:648] HybridClock initialized: now 1781967216786752 us; error 33 us; skew 500 ppm
I20260620 14:53:36.787174 24112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:36 dist-test-slave-3mqx krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967216, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:36.790377 24112 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:36.792028 24112 webserver.cc:460] Webserver started at http://127.21.159.194:42111/ using document root <none> and password file <none>
I20260620 14:53:36.792709 24112 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:36.792858 24112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:36.793174 24112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:36.796255 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "77ceadd02f0c419ea39d1e4201750b6a"
format_stamp: "Formatted at 2026-06-20 14:53:36 on dist-test-slave-3mqx"
server_key: "0d92542aff4d50bead53798ad5a04a7e"
server_key_iv: "bfe7d8db12e7b22dddb3b15d503ffd4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:36.797052 24112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "77ceadd02f0c419ea39d1e4201750b6a"
format_stamp: "Formatted at 2026-06-20 14:53:36 on dist-test-slave-3mqx"
server_key: "0d92542aff4d50bead53798ad5a04a7e"
server_key_iv: "bfe7d8db12e7b22dddb3b15d503ffd4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:36.802486 24112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 14:53:36.806226 24127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:36.807806 24112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:36.807992 24112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "77ceadd02f0c419ea39d1e4201750b6a"
format_stamp: "Formatted at 2026-06-20 14:53:36 on dist-test-slave-3mqx"
server_key: "0d92542aff4d50bead53798ad5a04a7e"
server_key_iv: "bfe7d8db12e7b22dddb3b15d503ffd4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:36.808211 24112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:36.831106 24112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:36.835976 24112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:36.836283 24112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:36.839150 24112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:36.839286 24112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:36.839422 24112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:36.839520 24112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:53:36 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967216, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:36.865571 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.840017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:53277 (local address 127.21.159.254:39611)
0620 14:53:36.840210 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:36.840216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:36.841642 (+ 1426us) server_negotiation.cc:407] Connection header received
0620 14:53:36.843857 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:36.843864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:36.843954 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:36.844099 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:36.846459 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.847524 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.849547 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.850107 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.850256 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.856407 (+ 6151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:36.856414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:36.856420 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:36.856469 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:36.860307 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.861605 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.861614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.861620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.861790 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.862450 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.862454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.862456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.862862 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:36.863033 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:36.864604 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0620 14:53:36.864950 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:53:36.867722 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.840623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39611 (local address 127.21.159.194:53277)
0620 14:53:36.841463 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:36.841510 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:36.843486 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:36.844335 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:36.844343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:36.845158 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:36.846236 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.846270 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.847711 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.847720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.849367 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.849381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.850393 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.850401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.850795 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.851790 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:36.851819 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:36.856053 (+ 4234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:36.860493 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.860505 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.860530 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.861368 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.861915 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.861923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.861927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.862284 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.863159 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:36.863168 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:36.863779 (+ 611us) client_negotiation.cc:769] Sending connection context
0620 14:53:36.864203 (+ 424us) client_negotiation.cc:240] Negotiation successful
0620 14:53:36.866181 (+ 1978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":375,"threads_started":1}
I20260620 14:53:36.891227 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.874595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:49695 (local address 127.21.159.254:39611)
0620 14:53:36.874811 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:36.874818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:36.875511 (+ 693us) server_negotiation.cc:407] Connection header received
0620 14:53:36.875916 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:36.875922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:36.876016 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:36.876145 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:36.877693 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.878793 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.880965 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.881641 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.881797 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.884437 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:36.884447 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:36.884451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:36.884484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:36.887648 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.888756 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.888762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.888765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.888915 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:36.889388 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:36.889396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:36.889402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:36.889881 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:36.890040 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:36.890442 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 14:53:36.890713 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:53:36.891294 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:36.874623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39611 (local address 127.21.159.194:49695)
0620 14:53:36.875383 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:36.875412 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:36.875708 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:36.876170 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:36.876174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:36.877013 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:36.877506 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.877520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.878934 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.878939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.880746 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:36.880766 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:36.881841 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:36.881846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:36.882002 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:36.882853 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:36.882880 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:36.884219 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:36.887796 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.887801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.887805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.888590 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.889034 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:36.889042 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:36.889049 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:36.889233 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:36.890011 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:36.890015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:36.890176 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 14:53:36.890282 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:53:36.890733 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":446,"threads_started":1}
I20260620 14:53:36.892318 24112 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:45945
I20260620 14:53:36.892352 24246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:45945 every 8 connection(s)
I20260620 14:53:36.893724 24112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:36.895926 24247 heartbeater.cc:347] Connected to a master server at 127.21.159.254:39611
I20260620 14:53:36.896118 24247 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:36.896710 24247 heartbeater.cc:511] Master 127.21.159.254:39611 requested a full tablet report, sending...
I20260620 14:53:36.897768 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 24112
I20260620 14:53:36.897961 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:36.898099 23901 ts_manager.cc:194] Registered new tserver with Master: 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945)
I20260620 14:53:36.898463 22143 external_mini_cluster.cc:1428] Setting key 27b87e00d5677a94877953a0ff8a6054
I20260620 14:53:36.898948 23901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:53277
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:36.940554 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39611
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:37.278506 24251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:37.278834 24251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:37.278945 24251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:37.286572 24251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:37.286700 24251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:37.286796 24251 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:53:37.286851 24251 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:53:37.286921 24251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:37.297080 24251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:39611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:37.298518 24251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:37.299940 24251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:37.314415 24257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:37.314518 24256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:37.314630 24259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:37.316038 24251 server_base.cc:1034] running on GCE node
I20260620 14:53:37.316824 24251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:37.318050 24251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:37.319310 24251 hybrid_clock.cc:648] HybridClock initialized: now 1781967217319264 us; error 88 us; skew 500 ppm
I20260620 14:53:37.319687 24251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:37 dist-test-slave-3mqx krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967217, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:37.323467 24251 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:37.325279 24251 webserver.cc:460] Webserver started at http://127.21.159.195:34223/ using document root <none> and password file <none>
I20260620 14:53:37.326066 24251 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:37.326231 24251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:37.326586 24251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:37.330075 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "0c85f8031a4040038b5eb20dbd2bca6e"
format_stamp: "Formatted at 2026-06-20 14:53:37 on dist-test-slave-3mqx"
server_key: "3aaeab8e456c48f620032ac240657fae"
server_key_iv: "75de68de6b616f4ffe614f5f5a58089f"
server_key_version: "encryptionkey@0"
I20260620 14:53:37.330955 24251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "0c85f8031a4040038b5eb20dbd2bca6e"
format_stamp: "Formatted at 2026-06-20 14:53:37 on dist-test-slave-3mqx"
server_key: "3aaeab8e456c48f620032ac240657fae"
server_key_iv: "75de68de6b616f4ffe614f5f5a58089f"
server_key_version: "encryptionkey@0"
I20260620 14:53:37.337126 24251 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 14:53:37.341717 24266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:37.343492 24251 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:53:37.343690 24251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "0c85f8031a4040038b5eb20dbd2bca6e"
format_stamp: "Formatted at 2026-06-20 14:53:37 on dist-test-slave-3mqx"
server_key: "3aaeab8e456c48f620032ac240657fae"
server_key_iv: "75de68de6b616f4ffe614f5f5a58089f"
server_key_version: "encryptionkey@0"
I20260620 14:53:37.343954 24251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:37.352037 24108 heartbeater.cc:503] Master 127.21.159.254:39611 was elected leader, sending a full tablet report...
I20260620 14:53:37.387463 24251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:37.392052 24251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:37.392347 24251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:37.395107 24251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:37.395249 24251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:37.395383 24251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:37.395475 24251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:53:37 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967217, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:37.420905 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.397233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:39039 (local address 127.21.159.254:39611)
0620 14:53:37.398313 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.398321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.398347 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:53:37.400088 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.400094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.400216 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.400342 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:37.402728 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.403754 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.406766 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.407468 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.407659 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.413471 (+ 5812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:37.413477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:37.413481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:37.413518 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:37.416811 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.417995 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.418003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.418009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.418191 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.418846 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.418849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.418852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.419271 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:37.419425 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.420284 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.420525 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":838,"threads_started":1}
I20260620 14:53:37.421999 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.396621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39611 (local address 127.21.159.195:39039)
0620 14:53:37.397470 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.397529 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.399771 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.400660 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.400680 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.401539 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:37.402456 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.402489 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.403997 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.404004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.406490 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.406508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.407786 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.407796 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.408083 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.409000 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:37.409035 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:37.413090 (+ 4055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:37.417046 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:37.417057 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:37.417082 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:37.417776 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:37.418305 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:37.418310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:37.418320 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:37.418666 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:37.419497 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:37.419504 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:37.419872 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.420189 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.420769 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":359,"threads_started":1}
I20260620 14:53:37.445842 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.429505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:43237 (local address 127.21.159.254:39611)
0620 14:53:37.429690 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.429695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.430112 (+ 417us) server_negotiation.cc:407] Connection header received
0620 14:53:37.430362 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.430367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.430468 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.430591 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:37.432285 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.433364 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.435667 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.436208 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.436330 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.439281 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:37.439305 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:37.439313 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:37.439373 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:37.442668 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.443761 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.443768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.443774 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.443916 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.444342 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.444347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.444353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.444644 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:37.444798 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.445242 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.445485 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:53:37.445976 24342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.429236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39611 (local address 127.21.159.195:43237)
0620 14:53:37.429973 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.430001 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.430202 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.430684 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.430691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.431502 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:37.432039 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.432065 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.433538 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.433544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.435448 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.435466 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.436422 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.436431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.436683 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.437492 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:37.437529 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:37.438938 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:37.442891 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:37.442896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:37.442901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:37.443611 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:37.444062 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:37.444068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:37.444071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:37.444218 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:37.444763 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:37.444767 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:37.444986 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.445120 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.445497 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":348,"threads_started":1}
I20260620 14:53:37.455747 24251 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:43543
I20260620 14:53:37.455787 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:43543 every 8 connection(s)
I20260620 14:53:37.457115 24251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:37.459640 24387 heartbeater.cc:347] Connected to a master server at 127.21.159.254:39611
I20260620 14:53:37.459930 24387 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:37.460713 24387 heartbeater.cc:511] Master 127.21.159.254:39611 requested a full tablet report, sending...
I20260620 14:53:37.460871 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 24251
I20260620 14:53:37.461047 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:37.461509 22143 external_mini_cluster.cc:1428] Setting key 108481a46f4662dc0a2900e86a4f5584
I20260620 14:53:37.462306 23901 ts_manager.cc:194] Registered new tserver with Master: 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543)
I20260620 14:53:37.463258 23901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:39039
I20260620 14:53:37.464524 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:53:37 dist-test-slave-3mqx krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:53:37 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:37.513548 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.498880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.21.159.254:39611)
0620 14:53:37.499144 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.499152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.499385 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:53:37.499627 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.499632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.499710 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.499819 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:37.501296 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.502027 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.503464 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.503872 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.504010 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.508240 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:37.508246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:37.508249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:37.508287 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:37.510750 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.511575 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.511579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.511581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.511711 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.512105 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.512114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.512116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.512396 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:37.512665 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.512973 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.513194 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 14:53:37.518148 23900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:37.526420 23900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:37.550580 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.540071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.21.159.194:45945)
0620 14:53:37.541628 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.541638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.541679 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:53:37.541789 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.541797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.542068 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.542920 (+ 852us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:37.543140 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.544670 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.547702 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.549242 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.549419 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.549599 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.550027 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.550217 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1396,"thread_start_us":2192,"threads_started":1}
I20260620 14:53:37.550813 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.539898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45945 (local address 127.0.0.1:53718)
0620 14:53:37.540750 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.540801 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.541024 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.542314 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.542323 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.542358 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:37.542755 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.542765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.544839 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.544850 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.547554 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.547565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.549488 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.549497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.549732 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.549801 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.549974 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.550357 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":83,"thread_start_us":307,"threads_started":1}
I20260620 14:53:37.555270 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.540553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43543 (local address 127.0.0.1:54048)
0620 14:53:37.544777 (+ 4224us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.544799 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.544940 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.546120 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.546129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.546164 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:37.546586 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.546597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.549671 (+ 3074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.549677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.552171 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.552187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.554424 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.554431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.554583 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.554618 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.554718 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.554864 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4082,"mutex_wait_us":51,"thread_start_us":6415,"threads_started":1}
I20260620 14:53:37.555356 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.541082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.21.159.195:43543)
0620 14:53:37.544030 (+ 2948us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.544039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.545093 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 14:53:37.545206 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.545214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.545562 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.545814 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:37.546840 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.548651 (+ 1811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.552365 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.553921 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.554117 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.554286 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.554793 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.554973 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2781,"thread_start_us":2603,"threads_started":1}
I20260620 14:53:37.555487 24177 tablet_service.cc:1467] Processing CreateTablet for tablet 56cf89e4f3bc42bd893973efa860db74 (DEFAULT_TABLE table=test-table [id=1bced7a0f52341119d6384c678968900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:37.556686 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.541040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:36747 (local address 127.0.0.1:50100)
0620 14:53:37.547293 (+ 6253us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.547312 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.547441 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.548076 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.548081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.548106 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:37.548437 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.548447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.552347 (+ 3900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.552350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.553993 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.554006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.555983 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.555987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.556107 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.556127 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.556223 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.556362 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6057,"thread_start_us":3183,"threads_started":1}
I20260620 14:53:37.556730 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.541592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.21.159.193:36747)
0620 14:53:37.543505 (+ 1913us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.543514 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.547442 (+ 3928us) server_negotiation.cc:407] Connection header received
0620 14:53:37.547588 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.547596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.547849 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.548070 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:37.548668 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.550165 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.554182 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.555466 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.555613 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.555816 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.556269 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.556434 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1764,"thread_start_us":1639,"threads_started":1}
I20260620 14:53:37.557107 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cf89e4f3bc42bd893973efa860db74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:37.559602 24317 tablet_service.cc:1467] Processing CreateTablet for tablet 56cf89e4f3bc42bd893973efa860db74 (DEFAULT_TABLE table=test-table [id=1bced7a0f52341119d6384c678968900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:37.560158 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 56cf89e4f3bc42bd893973efa860db74 (DEFAULT_TABLE table=test-table [id=1bced7a0f52341119d6384c678968900]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:37.561193 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cf89e4f3bc42bd893973efa860db74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:37.561417 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cf89e4f3bc42bd893973efa860db74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:37.577224 24407 tablet_bootstrap.cc:492] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Bootstrap starting.
I20260620 14:53:37.579547 24406 tablet_bootstrap.cc:492] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Bootstrap starting.
I20260620 14:53:37.579862 24408 tablet_bootstrap.cc:492] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Bootstrap starting.
I20260620 14:53:37.581293 24407 tablet_bootstrap.cc:654] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:37.582782 24407 log.cc:824] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:37.583293 24406 tablet_bootstrap.cc:654] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:37.583739 24408 tablet_bootstrap.cc:654] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:37.584631 24406 log.cc:824] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:37.585027 24408 log.cc:824] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:37.586557 24407 tablet_bootstrap.cc:492] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: No bootstrap required, opened a new log
I20260620 14:53:37.586859 24407 ts_tablet_manager.cc:1397] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 14:53:37.588379 24408 tablet_bootstrap.cc:492] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: No bootstrap required, opened a new log
I20260620 14:53:37.588670 24408 ts_tablet_manager.cc:1397] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 14:53:37.588716 24406 tablet_bootstrap.cc:492] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: No bootstrap required, opened a new log
I20260620 14:53:37.588968 24406 ts_tablet_manager.cc:1397] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 14:53:37.597354 24407 raft_consensus.cc:348] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.597966 24407 raft_consensus.cc:374] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:37.598083 24407 raft_consensus.cc:729] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539a55a32a4894b6e4ed891ae58622, State: Initialized, Role: FOLLOWER
I20260620 14:53:37.598963 24407 consensus_queue.cc:260] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.600291 24408 raft_consensus.cc:348] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.600294 24406 raft_consensus.cc:348] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.600754 24406 raft_consensus.cc:374] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:37.600755 24408 raft_consensus.cc:374] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:37.600847 24406 raft_consensus.cc:729] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ceadd02f0c419ea39d1e4201750b6a, State: Initialized, Role: FOLLOWER
I20260620 14:53:37.600871 24408 raft_consensus.cc:729] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c85f8031a4040038b5eb20dbd2bca6e, State: Initialized, Role: FOLLOWER
I20260620 14:53:37.600943 24407 ts_tablet_manager.cc:1428] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 14:53:37.601536 24408 consensus_queue.cc:260] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.601665 24406 consensus_queue.cc:260] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.602798 24387 heartbeater.cc:503] Master 127.21.159.254:39611 was elected leader, sending a full tablet report...
I20260620 14:53:37.603636 24408 ts_tablet_manager.cc:1428] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 14:53:37.603724 24247 heartbeater.cc:503] Master 127.21.159.254:39611 was elected leader, sending a full tablet report...
I20260620 14:53:37.605361 24406 ts_tablet_manager.cc:1428] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
W20260620 14:53:37.646159 24248 tablet.cc:2369] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:37.684487 24412 raft_consensus.cc:482] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:37.684880 24412 raft_consensus.cc:504] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.686933 24412 leader_election.cc:288] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945), 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543)
I20260620 14:53:37.694887 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.688212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:50531 (local address 127.21.159.195:43543)
0620 14:53:37.688384 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.688390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.688674 (+ 284us) server_negotiation.cc:407] Connection header received
0620 14:53:37.688912 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.688917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.688997 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.689102 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:37.689812 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.690820 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.692755 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.693821 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.694007 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.694089 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.694416 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.694564 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:53:37.694957 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.687455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43543 (local address 127.21.159.193:50531)
0620 14:53:37.688582 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.688601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.688764 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.689137 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.689142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.689168 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:37.689628 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.689639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.690965 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.690970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.692585 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.692602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.694047 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.694053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.694233 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.694268 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.694365 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.694519 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":7,"thread_start_us":909,"threads_started":1}
I20260620 14:53:37.695842 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.687582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:44035 (local address 127.21.159.194:45945)
0620 14:53:37.687984 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.687991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.688313 (+ 322us) server_negotiation.cc:407] Connection header received
0620 14:53:37.688720 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.688726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.688805 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.688907 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:37.689517 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.690596 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.693182 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.694381 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.694547 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.694615 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.695444 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.695600 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260620 14:53:37.696086 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.687415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45945 (local address 127.21.159.193:44035)
0620 14:53:37.688173 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.688199 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.688454 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.688934 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.688939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.688961 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:37.689373 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.689384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.690736 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.690743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.693009 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.693024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.694835 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.694843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.695243 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.695279 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.695399 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.695543 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":5,"thread_start_us":416,"threads_started":1}
I20260620 14:53:37.696185 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cf89e4f3bc42bd893973efa860db74" candidate_uuid: "2f539a55a32a4894b6e4ed891ae58622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" is_pre_election: true
I20260620 14:53:37.696813 24341 raft_consensus.cc:2454] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539a55a32a4894b6e4ed891ae58622 in term 0.
I20260620 14:53:37.697273 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cf89e4f3bc42bd893973efa860db74" candidate_uuid: "2f539a55a32a4894b6e4ed891ae58622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ceadd02f0c419ea39d1e4201750b6a" is_pre_election: true
I20260620 14:53:37.697561 23991 leader_election.cc:302] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 2f539a55a32a4894b6e4ed891ae58622; no voters:
I20260620 14:53:37.697758 24197 raft_consensus.cc:2454] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539a55a32a4894b6e4ed891ae58622 in term 0.
I20260620 14:53:37.698053 24412 raft_consensus.cc:2781] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:37.698208 24412 raft_consensus.cc:482] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:37.698330 24412 raft_consensus.cc:3037] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:37.700680 24412 raft_consensus.cc:504] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.701331 24412 leader_election.cc:288] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [CANDIDATE]: Term 1 election: Requested vote from peers 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945), 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543)
I20260620 14:53:37.701856 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cf89e4f3bc42bd893973efa860db74" candidate_uuid: "2f539a55a32a4894b6e4ed891ae58622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ceadd02f0c419ea39d1e4201750b6a"
I20260620 14:53:37.701900 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cf89e4f3bc42bd893973efa860db74" candidate_uuid: "2f539a55a32a4894b6e4ed891ae58622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c85f8031a4040038b5eb20dbd2bca6e"
I20260620 14:53:37.702082 24197 raft_consensus.cc:3037] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:37.702124 24341 raft_consensus.cc:3037] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:37.704388 24197 raft_consensus.cc:2454] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539a55a32a4894b6e4ed891ae58622 in term 1.
I20260620 14:53:37.704872 23990 leader_election.cc:302] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539a55a32a4894b6e4ed891ae58622, 77ceadd02f0c419ea39d1e4201750b6a; no voters:
I20260620 14:53:37.705170 24412 raft_consensus.cc:2781] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:37.705421 24341 raft_consensus.cc:2454] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539a55a32a4894b6e4ed891ae58622 in term 1.
I20260620 14:53:37.706243 24412 raft_consensus.cc:686] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539a55a32a4894b6e4ed891ae58622, State: Running, Role: LEADER
I20260620 14:53:37.706744 24412 consensus_queue.cc:237] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
W20260620 14:53:37.708866 24388 tablet.cc:2369] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:37.712128 23900 catalog_manager.cc:5495] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193). New cstate: current_term: 1 leader_uuid: "2f539a55a32a4894b6e4ed891ae58622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:53:37 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967215, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:37.785188 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.769937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39611 (local address 127.0.0.1:48092)
0620 14:53:37.770598 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:37.770616 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:37.770848 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:37.771297 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:37.771302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:37.772133 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:37.772432 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.772443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.773571 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.773579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.774631 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:37.774644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.775547 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.775554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:37.775733 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.776551 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:37.776571 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:37.779114 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:37.782532 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:37.782548 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:37.782555 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:37.783152 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:37.783527 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:37.783531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:37.783534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:37.783669 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:37.784212 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:37.784219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:37.784394 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:53:37.784571 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 14:53:37.784823 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":369,"threads_started":1}
I20260620 14:53:37.785283 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.769849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.21.159.254:39611)
0620 14:53:37.770057 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.770066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.770720 (+ 654us) server_negotiation.cc:407] Connection header received
0620 14:53:37.771047 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.771093 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.771156 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.771297 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:37.772605 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.773432 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.774786 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.775327 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.775501 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.779336 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:37.779342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:37.779346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:37.779387 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:37.782335 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.783281 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.783285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.783287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.783413 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.783770 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.783773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.783775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.784052 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:37.784203 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.784664 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.784945 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":77}
I20260620 14:53:37.791453 23900 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:48092:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:53:37.792459 23900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:37.799935 24177 tablet_service.cc:1467] Processing CreateTablet for tablet 37a1f8aee1aa4cc183b702f2b04509fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80b910373e274cbba2a8b3c29a211d86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:53:37.799901 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 37a1f8aee1aa4cc183b702f2b04509fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80b910373e274cbba2a8b3c29a211d86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:53:37.800942 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a1f8aee1aa4cc183b702f2b04509fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:37.800956 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a1f8aee1aa4cc183b702f2b04509fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:37.806602 24406 tablet_bootstrap.cc:492] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: Bootstrap starting.
I20260620 14:53:37.806267 24317 tablet_service.cc:1467] Processing CreateTablet for tablet 37a1f8aee1aa4cc183b702f2b04509fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80b910373e274cbba2a8b3c29a211d86]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:53:37.807163 24407 tablet_bootstrap.cc:492] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: Bootstrap starting.
I20260620 14:53:37.807372 24317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a1f8aee1aa4cc183b702f2b04509fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:37.808409 24406 tablet_bootstrap.cc:654] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:37.808873 24407 tablet_bootstrap.cc:654] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:37.811939 24407 tablet_bootstrap.cc:492] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: No bootstrap required, opened a new log
I20260620 14:53:37.812110 24407 ts_tablet_manager.cc:1397] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:53:37.812626 24406 tablet_bootstrap.cc:492] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: No bootstrap required, opened a new log
I20260620 14:53:37.812775 24406 ts_tablet_manager.cc:1397] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260620 14:53:37.812988 24407 raft_consensus.cc:348] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.813248 24407 raft_consensus.cc:374] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:37.813344 24407 raft_consensus.cc:729] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539a55a32a4894b6e4ed891ae58622, State: Initialized, Role: FOLLOWER
I20260620 14:53:37.813621 24407 consensus_queue.cc:260] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.813627 24406 raft_consensus.cc:348] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.814256 24407 ts_tablet_manager.cc:1428] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:37.814960 24408 tablet_bootstrap.cc:492] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: Bootstrap starting.
I20260620 14:53:37.814100 24412 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:37.815521 24412 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:37.813860 24406 raft_consensus.cc:374] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:37.815804 24406 raft_consensus.cc:729] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ceadd02f0c419ea39d1e4201750b6a, State: Initialized, Role: FOLLOWER
I20260620 14:53:37.816116 24406 consensus_queue.cc:260] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.816653 24414 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:37.816983 24408 tablet_bootstrap.cc:654] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:37.817088 24414 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:37.818310 24406 ts_tablet_manager.cc:1428] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260620 14:53:37.819931 24408 tablet_bootstrap.cc:492] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: No bootstrap required, opened a new log
I20260620 14:53:37.820108 24408 ts_tablet_manager.cc:1397] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:53:37.820814 24408 raft_consensus.cc:348] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.821019 24408 raft_consensus.cc:374] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:37.821126 24408 raft_consensus.cc:729] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c85f8031a4040038b5eb20dbd2bca6e, State: Initialized, Role: FOLLOWER
I20260620 14:53:37.821403 24408 consensus_queue.cc:260] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:37.822031 24408 ts_tablet_manager.cc:1428] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:37.821870 24413 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:37.822263 24413 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:53:37.843595 24109 tablet.cc:2369] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:38.007205 24414 raft_consensus.cc:482] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:38.007587 24414 raft_consensus.cc:504] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.009953 24414 leader_election.cc:288] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543)
I20260620 14:53:38.021531 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.010769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:37467 (local address 127.21.159.193:36747)
0620 14:53:38.011339 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.011349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.012464 (+ 1115us) server_negotiation.cc:407] Connection header received
0620 14:53:38.012627 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.012631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.012694 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.012780 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:38.013500 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.014553 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.018428 (+ 3875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.019781 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.019976 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.020056 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.021106 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.021267 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260620 14:53:38.021531 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.011572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:57141 (local address 127.21.159.195:43543)
0620 14:53:38.011828 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.011837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.011862 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:53:38.011952 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.011961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.012053 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.012161 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:38.013304 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.014538 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.018423 (+ 3885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.019781 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.019977 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.020056 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.021106 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.021267 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 14:53:38.021745 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.010319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43543 (local address 127.21.159.194:57141)
0620 14:53:38.011460 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.011488 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.011665 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.012348 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.012354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.012390 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:38.012921 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.012940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.014717 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.014723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.018124 (+ 3401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.018153 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.020336 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.020341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.020719 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.020864 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.020986 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.021240 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":426,"thread_start_us":1625,"threads_started":1}
I20260620 14:53:38.022094 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.010365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:36747 (local address 127.21.159.194:37467)
0620 14:53:38.012329 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.012348 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.012478 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.012845 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.012853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.012886 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:38.013307 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.013321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.014717 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.014723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.018124 (+ 3401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.018153 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.020322 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.020331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.020714 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.020864 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.020986 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.021239 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1804,"mutex_wait_us":416,"thread_start_us":744,"threads_started":1}
I20260620 14:53:38.022539 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a1f8aee1aa4cc183b702f2b04509fe" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" is_pre_election: true
I20260620 14:53:38.022811 24341 raft_consensus.cc:2454] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 0.
I20260620 14:53:38.023038 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a1f8aee1aa4cc183b702f2b04509fe" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622" is_pre_election: true
I20260620 14:53:38.023396 24131 leader_election.cc:302] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 77ceadd02f0c419ea39d1e4201750b6a; no voters:
I20260620 14:53:38.023526 24058 raft_consensus.cc:2454] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 0.
I20260620 14:53:38.024024 24414 raft_consensus.cc:2781] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:38.024188 24414 raft_consensus.cc:482] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:38.024295 24414 raft_consensus.cc:3037] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:38.026712 24414 raft_consensus.cc:504] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.027455 24414 leader_election.cc:288] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 election: Requested vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543)
I20260620 14:53:38.028074 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a1f8aee1aa4cc183b702f2b04509fe" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c85f8031a4040038b5eb20dbd2bca6e"
I20260620 14:53:38.028091 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37a1f8aee1aa4cc183b702f2b04509fe" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622"
I20260620 14:53:38.028342 24058 raft_consensus.cc:3037] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:38.028342 24341 raft_consensus.cc:3037] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:38.031438 24341 raft_consensus.cc:2454] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 1.
I20260620 14:53:38.031456 24058 raft_consensus.cc:2454] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 1.
I20260620 14:53:38.031955 24131 leader_election.cc:302] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 77ceadd02f0c419ea39d1e4201750b6a; no voters:
I20260620 14:53:38.032361 24414 raft_consensus.cc:2781] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:38.032806 24414 raft_consensus.cc:686] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [term 1 LEADER]: Becoming Leader. State: Replica: 77ceadd02f0c419ea39d1e4201750b6a, State: Running, Role: LEADER
I20260620 14:53:38.033316 24414 consensus_queue.cc:237] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.037528 24430 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: TxnStatusTablet state changed. Reason: New leader 77ceadd02f0c419ea39d1e4201750b6a. Latest consensus state: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:38.037905 24430 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:53:38.038621 24432 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:53:38.038578 23900 catalog_manager.cc:5495] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194). New cstate: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } health_report { overall_health: UNKNOWN } } }
I20260620 14:53:38.038960 24432 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:53:38.101636 24412 consensus_queue.cc:1035] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:38.110267 24417 consensus_queue.cc:1035] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:38.120492 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.107816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45945 (local address 127.0.0.1:53730)
0620 14:53:38.108424 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.108447 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.108594 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.109139 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.109143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.109157 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:38.109385 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.109401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.113513 (+ 4112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.113522 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.115834 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.115859 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.117564 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.117574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.117930 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.118053 (+ 123us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.119838 (+ 1785us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.119865 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.119951 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.120155 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410}
I20260620 14:53:38.121140 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.107879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.21.159.194:45945)
0620 14:53:38.108041 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.108048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.108799 (+ 751us) server_negotiation.cc:407] Connection header received
0620 14:53:38.108898 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.108902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.108972 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.109114 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:38.109584 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.111114 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.116129 (+ 5015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.117143 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.117344 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.118372 (+ 1028us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.119592 (+ 1220us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.119765 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.120736 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.120829 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:53:38.129544 24341 raft_consensus.cc:1261] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Refusing update from remote peer 77ceadd02f0c419ea39d1e4201750b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:38.130103 24058 raft_consensus.cc:1261] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Refusing update from remote peer 77ceadd02f0c419ea39d1e4201750b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:38.130367 24430 consensus_queue.cc:1035] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:38.130896 24414 consensus_queue.cc:1035] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:38.138386 24417 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: TxnStatusTablet state changed. Reason: New leader 77ceadd02f0c419ea39d1e4201750b6a. Latest consensus state: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:38.138734 24417 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:38.140880 24437 mvcc.cc:205] Tried to move back new op lower bound from 7298937725449175040 to 7298937725067751424. Current Snapshot: MvccSnapshot[applied={T|T < 7298937725449175040}]
I20260620 14:53:38.144266 24436 mvcc.cc:205] Tried to move back new op lower bound from 7298937725449175040 to 7298937725067751424. Current Snapshot: MvccSnapshot[applied={T|T < 7298937725449175040}]
I20260620 14:53:38.147714 24430 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:38.148061 24430 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:53:38.153501 24413 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: TxnStatusTablet state changed. Reason: New leader 77ceadd02f0c419ea39d1e4201750b6a. Latest consensus state: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:38.153832 24413 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:38.155650 24414 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:38.155948 24414 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 77ceadd02f0c419ea39d1e4201750b6a: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:53:38.156010 24417 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:38.156287 24417 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 2f539a55a32a4894b6e4ed891ae58622: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:38.158667 24439 mvcc.cc:205] Tried to move back new op lower bound from 7298937725449175040 to 7298937725067751424. Current Snapshot: MvccSnapshot[applied={T|T < 7298937725449175040}]
I20260620 14:53:38.164821 24413 tablet_replica.cc:439] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } } }
I20260620 14:53:38.165118 24413 tablet_replica.cc:442] T 37a1f8aee1aa4cc183b702f2b04509fe P 0c85f8031a4040038b5eb20dbd2bca6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:53:38.173873 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.167160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.21.159.193:36747)
0620 14:53:38.167326 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.167333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.167642 (+ 309us) server_negotiation.cc:407] Connection header received
0620 14:53:38.167757 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.167762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.167865 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.168042 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:38.168522 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.169618 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.171242 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.171922 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.172057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.172483 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.173232 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.173383 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.173435 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.173525 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:53:38.192492 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.186724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:38025 (local address 127.21.159.194:45945)
0620 14:53:38.186915 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.186919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.187522 (+ 603us) server_negotiation.cc:407] Connection header received
0620 14:53:38.187751 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.187755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.187825 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.187920 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:38.188422 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.189215 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.190641 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.191137 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.191277 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.191747 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.191849 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.191944 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.192088 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.192171 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:53:38.192595 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.186706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45945 (local address 127.21.159.193:38025)
0620 14:53:38.187408 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.187424 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.187607 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.187947 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.187952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.187965 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:38.188266 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.188275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.189369 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.189377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.190483 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.190494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.191328 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.191337 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.191459 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.191527 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.191953 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.191966 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.192059 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.192198 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":400,"threads_started":1}
I20260620 14:53:38.239282 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.231334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:45483 (local address 127.21.159.193:36747)
0620 14:53:38.231524 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.231530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.232231 (+ 701us) server_negotiation.cc:407] Connection header received
0620 14:53:38.232597 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.232603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.232693 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.232790 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:38.233388 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.234431 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.236928 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.237443 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.237586 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.238215 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.238410 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.238508 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.238852 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.238942 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:53:38.240104 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.231354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:36747 (local address 127.21.159.194:45483)
0620 14:53:38.232108 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.232134 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.232264 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.232822 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.232828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.232841 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:38.233182 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.233193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.234560 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.234566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.236749 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.236770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.237786 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.237793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.237962 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.238025 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.238533 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.238546 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.239552 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.239724 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":365,"threads_started":1}
I20260620 14:53:38.291476 23899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:38.291801 23899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:38.296109 23899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Sending DeleteTablet for 3 replicas of tablet 56cf89e4f3bc42bd893973efa860db74
I20260620 14:53:38.297528 24317 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cf89e4f3bc42bd893973efa860db74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:38 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54048
I20260620 14:53:38.297724 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cf89e4f3bc42bd893973efa860db74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:38 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:50100
I20260620 14:53:38.298313 24177 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cf89e4f3bc42bd893973efa860db74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:38 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:53718
I20260620 14:53:38.304812 23899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:38.305833 23899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:38.311908 24459 tablet_replica.cc:330] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: stopping tablet replica
I20260620 14:53:38.313699 24459 raft_consensus.cc:2229] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:38.314337 24459 raft_consensus.cc:2258] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:38.315728 24037 tablet_service.cc:1467] Processing CreateTablet for tablet 14950cad576e447ab4226bc9faff79e9 (DEFAULT_TABLE table=test-table [id=000fb7c48c894350974c014c5d90478f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:38.316036 24176 tablet_service.cc:1467] Processing CreateTablet for tablet 14950cad576e447ab4226bc9faff79e9 (DEFAULT_TABLE table=test-table [id=000fb7c48c894350974c014c5d90478f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:38.316226 24037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14950cad576e447ab4226bc9faff79e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:38.316488 24176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14950cad576e447ab4226bc9faff79e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:38.316517 24316 tablet_service.cc:1467] Processing CreateTablet for tablet 14950cad576e447ab4226bc9faff79e9 (DEFAULT_TABLE table=test-table [id=000fb7c48c894350974c014c5d90478f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:38.317018 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14950cad576e447ab4226bc9faff79e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:38.322072 24457 tablet_replica.cc:330] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: stopping tablet replica
I20260620 14:53:38.322623 24457 raft_consensus.cc:2229] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:38.323029 24457 raft_consensus.cc:2258] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:38.325173 24457 ts_tablet_manager.cc:1905] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:38.333258 24457 ts_tablet_manager.cc:1918] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:38.333362 24461 tablet_bootstrap.cc:492] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: Bootstrap starting.
I20260620 14:53:38.333468 24457 log.cc:1196] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/56cf89e4f3bc42bd893973efa860db74
I20260620 14:53:38.334043 24457 ts_tablet_manager.cc:1939] T 56cf89e4f3bc42bd893973efa860db74 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting consensus metadata
I20260620 14:53:38.336030 24460 tablet_bootstrap.cc:492] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: Bootstrap starting.
I20260620 14:53:38.336189 23888 catalog_manager.cc:4841] TS 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543): tablet 56cf89e4f3bc42bd893973efa860db74 (table test-table [id=1bced7a0f52341119d6384c678968900]) successfully deleted
I20260620 14:53:38.339670 24459 ts_tablet_manager.cc:1905] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:38.343472 24462 tablet_bootstrap.cc:492] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: Bootstrap starting.
I20260620 14:53:38.345077 24458 tablet_replica.cc:330] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: stopping tablet replica
I20260620 14:53:38.347366 24461 tablet_bootstrap.cc:654] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:38.347728 24458 raft_consensus.cc:2229] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:38.348523 24458 raft_consensus.cc:2258] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:38.349869 24460 tablet_bootstrap.cc:654] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:38.350553 24458 ts_tablet_manager.cc:1905] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:38.356259 24460 tablet_bootstrap.cc:492] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: No bootstrap required, opened a new log
I20260620 14:53:38.356436 24460 ts_tablet_manager.cc:1397] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20260620 14:53:38.356868 24459 ts_tablet_manager.cc:1918] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:38.357497 24460 raft_consensus.cc:348] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.357733 24460 raft_consensus.cc:374] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:38.357825 24460 raft_consensus.cc:729] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c85f8031a4040038b5eb20dbd2bca6e, State: Initialized, Role: FOLLOWER
I20260620 14:53:38.358116 24460 consensus_queue.cc:260] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.358376 24462 tablet_bootstrap.cc:654] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:38.359815 24461 tablet_bootstrap.cc:492] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: No bootstrap required, opened a new log
I20260620 14:53:38.360052 24461 ts_tablet_manager.cc:1397] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.001s
I20260620 14:53:38.360170 24460 ts_tablet_manager.cc:1428] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:53:38.361063 24461 raft_consensus.cc:348] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.361500 24461 raft_consensus.cc:374] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:38.361665 24461 raft_consensus.cc:729] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539a55a32a4894b6e4ed891ae58622, State: Initialized, Role: FOLLOWER
I20260620 14:53:38.362010 24461 consensus_queue.cc:260] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.362748 24461 ts_tablet_manager.cc:1428] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:53:38.363796 24458 ts_tablet_manager.cc:1918] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:38.364006 24458 log.cc:1196] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/56cf89e4f3bc42bd893973efa860db74
I20260620 14:53:38.364694 24458 ts_tablet_manager.cc:1939] T 56cf89e4f3bc42bd893973efa860db74 P 2f539a55a32a4894b6e4ed891ae58622: Deleting consensus metadata
I20260620 14:53:38.364660 24459 log.cc:1196] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/56cf89e4f3bc42bd893973efa860db74
I20260620 14:53:38.366659 23887 catalog_manager.cc:4841] TS 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747): tablet 56cf89e4f3bc42bd893973efa860db74 (table test-table [id=1bced7a0f52341119d6384c678968900]) successfully deleted
I20260620 14:53:38.367440 24459 ts_tablet_manager.cc:1939] T 56cf89e4f3bc42bd893973efa860db74 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting consensus metadata
I20260620 14:53:38.369917 23887 catalog_manager.cc:4841] TS 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945): tablet 56cf89e4f3bc42bd893973efa860db74 (table test-table [id=1bced7a0f52341119d6384c678968900]) successfully deleted
I20260620 14:53:38.380115 24462 tablet_bootstrap.cc:492] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: No bootstrap required, opened a new log
I20260620 14:53:38.380337 24462 ts_tablet_manager.cc:1397] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.004s
I20260620 14:53:38.381263 24462 raft_consensus.cc:348] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.381542 24462 raft_consensus.cc:374] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:38.381667 24462 raft_consensus.cc:729] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ceadd02f0c419ea39d1e4201750b6a, State: Initialized, Role: FOLLOWER
I20260620 14:53:38.382017 24462 consensus_queue.cc:260] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.382845 24462 ts_tablet_manager.cc:1428] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 14:53:38.408582 24413 raft_consensus.cc:482] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:38.408835 24413 raft_consensus.cc:504] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.410562 24413 leader_election.cc:288] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945)
I20260620 14:53:38.423621 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.411024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:51425 (local address 127.21.159.193:36747)
0620 14:53:38.411328 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.411338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.411909 (+ 571us) server_negotiation.cc:407] Connection header received
0620 14:53:38.412228 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.412233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.412319 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.412390 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:38.413351 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.414958 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.420514 (+ 5556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.422196 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.422375 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.422472 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.423094 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.423262 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 14:53:38.424641 24466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.410805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:36747 (local address 127.21.159.195:51425)
0620 14:53:38.411806 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.411829 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.412029 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.412594 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.412600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.412642 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:38.413118 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.413136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.417067 (+ 3931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.417076 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.420225 (+ 3149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.420250 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.422638 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.422645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.422874 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.422906 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.424165 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.424316 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":732,"threads_started":1}
I20260620 14:53:38.425606 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14950cad576e447ab4226bc9faff79e9" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622" is_pre_election: true
I20260620 14:53:38.425968 24057 raft_consensus.cc:2454] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 0.
I20260620 14:53:38.426659 24269 leader_election.cc:302] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 2f539a55a32a4894b6e4ed891ae58622; no voters:
I20260620 14:53:38.427215 24413 raft_consensus.cc:2781] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:38.427338 24413 raft_consensus.cc:482] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:38.427423 24413 raft_consensus.cc:3037] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:38.430529 24413 raft_consensus.cc:504] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.431936 24413 leader_election.cc:288] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 election: Requested vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945)
I20260620 14:53:38.432320 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.412418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:42969 (local address 127.21.159.194:45945)
0620 14:53:38.412625 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.412631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.422048 (+ 9417us) server_negotiation.cc:407] Connection header received
0620 14:53:38.422567 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.422572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.422654 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.422715 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:38.423592 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.424755 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.427153 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.430885 (+ 3732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.431093 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.431205 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.431831 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.431985 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:53:38.432905 24057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14950cad576e447ab4226bc9faff79e9" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622"
I20260620 14:53:38.433080 24467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.412008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45945 (local address 127.21.159.195:42969)
0620 14:53:38.421932 (+ 9924us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.421957 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.422386 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.422894 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.422899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.422922 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:38.423393 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.423406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.424978 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.424983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.426936 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.426950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.431423 (+ 4473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.431430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.431631 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.431663 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.432559 (+ 896us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.432724 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9778,"thread_start_us":10071,"threads_started":1}
I20260620 14:53:38.433153 24057 raft_consensus.cc:3037] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:38.435324 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14950cad576e447ab4226bc9faff79e9" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ceadd02f0c419ea39d1e4201750b6a" is_pre_election: true
I20260620 14:53:38.435637 24197 raft_consensus.cc:2454] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 0.
I20260620 14:53:38.435792 24196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14950cad576e447ab4226bc9faff79e9" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ceadd02f0c419ea39d1e4201750b6a"
I20260620 14:53:38.436033 24196 raft_consensus.cc:3037] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:38.439505 24196 raft_consensus.cc:2454] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 1.
I20260620 14:53:38.439504 24057 raft_consensus.cc:2454] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 1.
I20260620 14:53:38.440018 24269 leader_election.cc:302] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 2f539a55a32a4894b6e4ed891ae58622; no voters:
I20260620 14:53:38.440385 24413 raft_consensus.cc:2781] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:38.440974 24413 raft_consensus.cc:686] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 LEADER]: Becoming Leader. State: Replica: 0c85f8031a4040038b5eb20dbd2bca6e, State: Running, Role: LEADER
I20260620 14:53:38.441491 24413 consensus_queue.cc:237] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:38.445443 23899 catalog_manager.cc:5495] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195). New cstate: current_term: 1 leader_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } health_report { overall_health: HEALTHY } } }
I20260620 14:53:38.484540 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.477048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.21.159.195:43543)
0620 14:53:38.477203 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.477207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.477221 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:53:38.477304 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.477307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.477381 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.477550 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:38.477963 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.479120 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.481064 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.481812 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.481926 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.482544 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.483348 (+ 804us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.483481 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.484109 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.484199 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:53:38.522543 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.516328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:44321 (local address 127.21.159.194:45945)
0620 14:53:38.516537 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.516545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.517227 (+ 682us) server_negotiation.cc:407] Connection header received
0620 14:53:38.517466 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.517471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.517557 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.517661 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:38.518175 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.518987 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.520481 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.520970 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.521125 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.521561 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.521685 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.521793 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.522106 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.522206 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:53:38.522619 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.516331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45945 (local address 127.21.159.195:44321)
0620 14:53:38.517100 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.517117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.517313 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.517689 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.517693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.517706 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:38.517996 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.518005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.519188 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.519194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.520323 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.520337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.521163 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.521170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.521317 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.521377 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.521959 (+ 582us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.521975 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.522080 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.522223 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":428,"threads_started":1}
I20260620 14:53:38.538009 24057 raft_consensus.cc:1261] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Refusing update from remote peer 0c85f8031a4040038b5eb20dbd2bca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:38.538008 24196 raft_consensus.cc:1261] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Refusing update from remote peer 0c85f8031a4040038b5eb20dbd2bca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:38.538810 24468 consensus_queue.cc:1035] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:38.539608 24413 consensus_queue.cc:1035] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:38.593613 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.586585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43543 (local address 127.21.159.194:34659)
0620 14:53:38.586806 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:38.586830 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:38.586958 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:38.587617 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:38.587621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:38.587634 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:38.587843 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.587857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.589307 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.589316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.590860 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:38.590880 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.592019 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.592025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:38.592237 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.592289 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.592796 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.592811 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:53:38.592917 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:53:38.593140 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:53:38.594343 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.586980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:34659 (local address 127.21.159.195:43543)
0620 14:53:38.587177 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.587184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.587203 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:53:38.587269 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.587273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.587357 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.587456 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:38.588024 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.589097 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.591081 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.591766 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.591886 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.592477 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:38.592630 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:38.592733 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.593938 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.594057 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 14:53:38.595526 24439 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:53:40.210382 23899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:40.210646 23899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:40.215476 23899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Sending DeleteTablet for 3 replicas of tablet 14950cad576e447ab4226bc9faff79e9
I20260620 14:53:40.216471 24316 tablet_service.cc:1514] Processing DeleteTablet for tablet 14950cad576e447ab4226bc9faff79e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54048
I20260620 14:53:40.216454 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 14950cad576e447ab4226bc9faff79e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:50100
I20260620 14:53:40.216523 24176 tablet_service.cc:1514] Processing DeleteTablet for tablet 14950cad576e447ab4226bc9faff79e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:53718
I20260620 14:53:40.231253 24501 tablet_replica.cc:330] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: stopping tablet replica
I20260620 14:53:40.231508 24502 tablet_replica.cc:330] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: stopping tablet replica
I20260620 14:53:40.231556 24501 raft_consensus.cc:2229] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:40.232178 24502 raft_consensus.cc:2229] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:40.232206 24501 raft_consensus.cc:2258] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.232446 24502 raft_consensus.cc:2258] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.233304 24502 ts_tablet_manager.cc:1905] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.234933 24501 ts_tablet_manager.cc:1905] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.236449 24503 tablet_replica.cc:330] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: stopping tablet replica
I20260620 14:53:40.236660 24503 raft_consensus.cc:2229] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:40.236907 24503 raft_consensus.cc:2258] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.239748 24502 ts_tablet_manager.cc:1918] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:53:40.239775 24503 ts_tablet_manager.cc:1905] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.239900 24502 log.cc:1196] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/14950cad576e447ab4226bc9faff79e9
I20260620 14:53:40.240402 24502 ts_tablet_manager.cc:1939] T 14950cad576e447ab4226bc9faff79e9 P 2f539a55a32a4894b6e4ed891ae58622: Deleting consensus metadata
I20260620 14:53:40.241693 23887 catalog_manager.cc:4841] TS 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747): tablet 14950cad576e447ab4226bc9faff79e9 (table test-table [id=000fb7c48c894350974c014c5d90478f]) successfully deleted
I20260620 14:53:40.244287 24501 ts_tablet_manager.cc:1918] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:53:40.244457 24501 log.cc:1196] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/14950cad576e447ab4226bc9faff79e9
I20260620 14:53:40.245007 24501 ts_tablet_manager.cc:1939] T 14950cad576e447ab4226bc9faff79e9 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting consensus metadata
I20260620 14:53:40.246150 23888 catalog_manager.cc:4841] TS 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543): tablet 14950cad576e447ab4226bc9faff79e9 (table test-table [id=000fb7c48c894350974c014c5d90478f]) successfully deleted
I20260620 14:53:40.246973 24503 ts_tablet_manager.cc:1918] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:53:40.247138 24503 log.cc:1196] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/14950cad576e447ab4226bc9faff79e9
I20260620 14:53:40.247603 24503 ts_tablet_manager.cc:1939] T 14950cad576e447ab4226bc9faff79e9 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting consensus metadata
I20260620 14:53:40.248864 23887 catalog_manager.cc:4841] TS 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945): tablet 14950cad576e447ab4226bc9faff79e9 (table test-table [id=000fb7c48c894350974c014c5d90478f]) successfully deleted
Jun 20 14:53:40 dist-test-slave-3mqx krb5kdc[23856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:53:40 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:40.282706 24511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.266616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.21.159.254:39611)
0620 14:53:40.267308 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.267314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.267341 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:53:40.267453 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.267458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.267545 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.267661 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:40.269046 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.269860 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.271221 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.271674 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.271819 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.276392 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:40.276398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:40.276401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:40.276442 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:40.279535 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:40.280332 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:40.280338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:40.280341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:40.280466 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:40.280853 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:40.280863 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:40.280865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:40.281137 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:40.281406 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.281720 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.281944 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":302,"threads_started":1}
I20260620 14:53:40.286590 23899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:40.287297 23899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:40.294096 24176 tablet_service.cc:1467] Processing CreateTablet for tablet 82dabe9f32094319861d8b98e0782d8d (DEFAULT_TABLE table=test-table [id=9121f73d580a414a92eb2f4fadc582f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.294169 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 82dabe9f32094319861d8b98e0782d8d (DEFAULT_TABLE table=test-table [id=9121f73d580a414a92eb2f4fadc582f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.294581 24176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82dabe9f32094319861d8b98e0782d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.294615 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82dabe9f32094319861d8b98e0782d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.296618 24316 tablet_service.cc:1467] Processing CreateTablet for tablet 82dabe9f32094319861d8b98e0782d8d (DEFAULT_TABLE table=test-table [id=9121f73d580a414a92eb2f4fadc582f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.297092 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82dabe9f32094319861d8b98e0782d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.300751 24512 tablet_bootstrap.cc:492] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: Bootstrap starting.
I20260620 14:53:40.301509 24513 tablet_bootstrap.cc:492] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: Bootstrap starting.
I20260620 14:53:40.302702 24512 tablet_bootstrap.cc:654] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.303702 24513 tablet_bootstrap.cc:654] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.304945 24514 tablet_bootstrap.cc:492] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: Bootstrap starting.
I20260620 14:53:40.306566 24513 tablet_bootstrap.cc:492] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: No bootstrap required, opened a new log
I20260620 14:53:40.306746 24513 ts_tablet_manager.cc:1397] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 14:53:40.306999 24514 tablet_bootstrap.cc:654] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.307446 24512 tablet_bootstrap.cc:492] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: No bootstrap required, opened a new log
I20260620 14:53:40.307605 24512 ts_tablet_manager.cc:1397] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 14:53:40.307466 24513 raft_consensus.cc:348] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.307715 24513 raft_consensus.cc:374] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.307811 24513 raft_consensus.cc:729] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539a55a32a4894b6e4ed891ae58622, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.308106 24513 consensus_queue.cc:260] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.308657 24512 raft_consensus.cc:348] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.309002 24512 raft_consensus.cc:374] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.309116 24512 raft_consensus.cc:729] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ceadd02f0c419ea39d1e4201750b6a, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.309485 24512 consensus_queue.cc:260] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.310472 24512 ts_tablet_manager.cc:1428] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:40.311946 24513 ts_tablet_manager.cc:1428] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 14:53:40.314767 24514 tablet_bootstrap.cc:492] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: No bootstrap required, opened a new log
I20260620 14:53:40.314940 24514 ts_tablet_manager.cc:1397] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260620 14:53:40.315896 24514 raft_consensus.cc:348] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.316190 24514 raft_consensus.cc:374] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.316323 24514 raft_consensus.cc:729] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c85f8031a4040038b5eb20dbd2bca6e, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.316589 24514 consensus_queue.cc:260] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.317188 24514 ts_tablet_manager.cc:1428] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:53:40.384179 24479 raft_consensus.cc:482] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:40.384418 24479 raft_consensus.cc:504] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.385224 24479 leader_election.cc:288] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543)
I20260620 14:53:40.385780 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dabe9f32094319861d8b98e0782d8d" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622" is_pre_election: true
I20260620 14:53:40.385825 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dabe9f32094319861d8b98e0782d8d" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" is_pre_election: true
I20260620 14:53:40.386042 24058 raft_consensus.cc:2454] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 0.
I20260620 14:53:40.386065 24341 raft_consensus.cc:2454] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 0.
I20260620 14:53:40.386615 24131 leader_election.cc:302] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 2f539a55a32a4894b6e4ed891ae58622, 77ceadd02f0c419ea39d1e4201750b6a; no voters:
I20260620 14:53:40.386968 24479 raft_consensus.cc:2781] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:40.387115 24479 raft_consensus.cc:482] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:40.387238 24479 raft_consensus.cc:3037] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.389778 24479 raft_consensus.cc:504] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.390411 24479 leader_election.cc:288] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 election: Requested vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543)
I20260620 14:53:40.390977 24341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dabe9f32094319861d8b98e0782d8d" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c85f8031a4040038b5eb20dbd2bca6e"
I20260620 14:53:40.390970 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82dabe9f32094319861d8b98e0782d8d" candidate_uuid: "77ceadd02f0c419ea39d1e4201750b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622"
I20260620 14:53:40.391239 24058 raft_consensus.cc:3037] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.391258 24341 raft_consensus.cc:3037] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.394593 24341 raft_consensus.cc:2454] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 1.
I20260620 14:53:40.395030 24058 raft_consensus.cc:2454] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ceadd02f0c419ea39d1e4201750b6a in term 1.
I20260620 14:53:40.395291 24131 leader_election.cc:302] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 77ceadd02f0c419ea39d1e4201750b6a; no voters:
I20260620 14:53:40.395946 24479 raft_consensus.cc:2781] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:40.396140 24479 raft_consensus.cc:686] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 1 LEADER]: Becoming Leader. State: Replica: 77ceadd02f0c419ea39d1e4201750b6a, State: Running, Role: LEADER
I20260620 14:53:40.396564 24479 consensus_queue.cc:237] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.400044 23900 catalog_manager.cc:5495] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194). New cstate: current_term: 1 leader_uuid: "77ceadd02f0c419ea39d1e4201750b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } health_report { overall_health: UNKNOWN } } }
I20260620 14:53:40.440872 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.435071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.21.159.194:45945)
0620 14:53:40.435707 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.435713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.435735 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:53:40.435793 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.435805 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.435866 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.435946 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:40.436535 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.437550 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.439036 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.439535 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.439635 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.440096 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:40.440202 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:40.440271 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.440434 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.440496 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":331,"threads_started":1}
I20260620 14:53:40.455199 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.450258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:51883 (local address 127.21.159.194:45945)
0620 14:53:40.450403 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.450407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.450816 (+ 409us) server_negotiation.cc:407] Connection header received
0620 14:53:40.450946 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.450949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.451006 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.451146 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:40.451578 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.452270 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.453547 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.453912 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.454000 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.454400 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:40.454490 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:40.454556 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.454758 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.454885 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":42}
I20260620 14:53:40.455583 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.450040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45945 (local address 127.21.159.194:51883)
0620 14:53:40.450718 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.450738 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:40.450850 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:40.451161 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:40.451165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:40.451174 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:53:40.451446 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.451454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.452385 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.452389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.453433 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.453444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.454061 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.454066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.454219 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.454255 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:53:40.454617 (+ 362us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:53:40.454630 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:53:40.454746 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:53:40.454941 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":50,"thread_start_us":411,"threads_started":1}
I20260620 14:53:40.464982 24058 raft_consensus.cc:1261] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Refusing update from remote peer 77ceadd02f0c419ea39d1e4201750b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:40.465659 24341 raft_consensus.cc:1261] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Refusing update from remote peer 77ceadd02f0c419ea39d1e4201750b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:40.465679 24479 consensus_queue.cc:1035] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:40.466344 24479 consensus_queue.cc:1035] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:40.525544 23899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:40.525818 23899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:40.529459 23899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Sending DeleteTablet for 3 replicas of tablet 82dabe9f32094319861d8b98e0782d8d
I20260620 14:53:40.530463 24316 tablet_service.cc:1514] Processing DeleteTablet for tablet 82dabe9f32094319861d8b98e0782d8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54048
I20260620 14:53:40.530462 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 82dabe9f32094319861d8b98e0782d8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:50100
I20260620 14:53:40.530763 24502 tablet_replica.cc:330] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: stopping tablet replica
I20260620 14:53:40.530829 24501 tablet_replica.cc:330] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: stopping tablet replica
I20260620 14:53:40.530982 24502 raft_consensus.cc:2229] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:40.531014 24501 raft_consensus.cc:2229] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:40.531208 24502 raft_consensus.cc:2258] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.531234 24501 raft_consensus.cc:2258] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.531819 24177 tablet_service.cc:1514] Processing DeleteTablet for tablet 82dabe9f32094319861d8b98e0782d8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:53718
I20260620 14:53:40.532116 24503 tablet_replica.cc:330] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: stopping tablet replica
I20260620 14:53:40.532311 24503 raft_consensus.cc:2229] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:40.532313 23899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:53:40.532567 24501 ts_tablet_manager.cc:1905] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.532848 24503 raft_consensus.cc:2258] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.533778 24502 ts_tablet_manager.cc:1905] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.534039 24503 ts_tablet_manager.cc:1905] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.539920 24501 ts_tablet_manager.cc:1918] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:40.540203 24501 log.cc:1196] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/82dabe9f32094319861d8b98e0782d8d
I20260620 14:53:40.540738 24501 ts_tablet_manager.cc:1939] T 82dabe9f32094319861d8b98e0782d8d P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting consensus metadata
I20260620 14:53:40.540957 24502 ts_tablet_manager.cc:1918] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:40.541134 24502 log.cc:1196] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/82dabe9f32094319861d8b98e0782d8d
W20260620 14:53:40.541121 23899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:40.542095 24502 ts_tablet_manager.cc:1939] T 82dabe9f32094319861d8b98e0782d8d P 2f539a55a32a4894b6e4ed891ae58622: Deleting consensus metadata
I20260620 14:53:40.542222 23888 catalog_manager.cc:4841] TS 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543): tablet 82dabe9f32094319861d8b98e0782d8d (table test-table [id=9121f73d580a414a92eb2f4fadc582f7]) successfully deleted
I20260620 14:53:40.542689 24503 ts_tablet_manager.cc:1918] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:53:40.542886 24503 log.cc:1196] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/82dabe9f32094319861d8b98e0782d8d
I20260620 14:53:40.543381 24503 ts_tablet_manager.cc:1939] T 82dabe9f32094319861d8b98e0782d8d P 77ceadd02f0c419ea39d1e4201750b6a: Deleting consensus metadata
I20260620 14:53:40.544152 23887 catalog_manager.cc:4841] TS 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747): tablet 82dabe9f32094319861d8b98e0782d8d (table test-table [id=9121f73d580a414a92eb2f4fadc582f7]) successfully deleted
I20260620 14:53:40.544517 23887 catalog_manager.cc:4841] TS 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945): tablet 82dabe9f32094319861d8b98e0782d8d (table test-table [id=9121f73d580a414a92eb2f4fadc582f7]) successfully deleted
I20260620 14:53:40.549150 24038 tablet_service.cc:1467] Processing CreateTablet for tablet 59c05bcc81fb4658a707af4c050a3a00 (DEFAULT_TABLE table=test-table [id=a73bd9c8306e41ecac80f029c3137a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.549238 24316 tablet_service.cc:1467] Processing CreateTablet for tablet 59c05bcc81fb4658a707af4c050a3a00 (DEFAULT_TABLE table=test-table [id=a73bd9c8306e41ecac80f029c3137a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.549314 24177 tablet_service.cc:1467] Processing CreateTablet for tablet 59c05bcc81fb4658a707af4c050a3a00 (DEFAULT_TABLE table=test-table [id=a73bd9c8306e41ecac80f029c3137a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.549612 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c05bcc81fb4658a707af4c050a3a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.549662 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c05bcc81fb4658a707af4c050a3a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.549788 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c05bcc81fb4658a707af4c050a3a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.556457 24514 tablet_bootstrap.cc:492] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: Bootstrap starting.
I20260620 14:53:40.556736 24513 tablet_bootstrap.cc:492] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: Bootstrap starting.
I20260620 14:53:40.557853 24512 tablet_bootstrap.cc:492] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: Bootstrap starting.
I20260620 14:53:40.558229 24514 tablet_bootstrap.cc:654] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.558326 24513 tablet_bootstrap.cc:654] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.559790 24512 tablet_bootstrap.cc:654] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.571354 24514 tablet_bootstrap.cc:492] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: No bootstrap required, opened a new log
I20260620 14:53:40.571650 24514 ts_tablet_manager.cc:1397] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260620 14:53:40.572461 24512 tablet_bootstrap.cc:492] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: No bootstrap required, opened a new log
I20260620 14:53:40.572685 24512 ts_tablet_manager.cc:1397] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260620 14:53:40.573750 24512 raft_consensus.cc:348] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.574100 24512 raft_consensus.cc:374] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.573920 24514 raft_consensus.cc:348] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.574206 24512 raft_consensus.cc:729] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ceadd02f0c419ea39d1e4201750b6a, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.574397 24514 raft_consensus.cc:374] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.574536 24514 raft_consensus.cc:729] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c85f8031a4040038b5eb20dbd2bca6e, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.574581 24512 consensus_queue.cc:260] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.574924 24514 consensus_queue.cc:260] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.575747 24514 ts_tablet_manager.cc:1428] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:53:40.576360 24512 ts_tablet_manager.cc:1428] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 14:53:40.579939 24513 tablet_bootstrap.cc:492] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: No bootstrap required, opened a new log
I20260620 14:53:40.580145 24513 ts_tablet_manager.cc:1397] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.012s
I20260620 14:53:40.580797 24513 raft_consensus.cc:348] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.581020 24513 raft_consensus.cc:374] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.581115 24513 raft_consensus.cc:729] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539a55a32a4894b6e4ed891ae58622, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.581360 24513 consensus_queue.cc:260] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.581810 24513 ts_tablet_manager.cc:1428] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 14:53:40.622243 24413 raft_consensus.cc:482] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:40.622510 24413 raft_consensus.cc:504] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.623409 24413 leader_election.cc:288] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945)
I20260620 14:53:40.623943 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c05bcc81fb4658a707af4c050a3a00" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622" is_pre_election: true
I20260620 14:53:40.624032 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c05bcc81fb4658a707af4c050a3a00" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ceadd02f0c419ea39d1e4201750b6a" is_pre_election: true
I20260620 14:53:40.624730 24056 raft_consensus.cc:2454] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 0.
I20260620 14:53:40.624760 24197 raft_consensus.cc:2454] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 0.
I20260620 14:53:40.625347 24269 leader_election.cc:302] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 77ceadd02f0c419ea39d1e4201750b6a; no voters:
I20260620 14:53:40.625676 24413 raft_consensus.cc:2781] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:40.625804 24413 raft_consensus.cc:482] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:40.625896 24413 raft_consensus.cc:3037] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.628968 24413 raft_consensus.cc:504] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.629622 24413 leader_election.cc:288] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 election: Requested vote from peers 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747), 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945)
I20260620 14:53:40.630194 24056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c05bcc81fb4658a707af4c050a3a00" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539a55a32a4894b6e4ed891ae58622"
I20260620 14:53:40.630295 24197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c05bcc81fb4658a707af4c050a3a00" candidate_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ceadd02f0c419ea39d1e4201750b6a"
I20260620 14:53:40.630438 24056 raft_consensus.cc:3037] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.630518 24197 raft_consensus.cc:3037] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.633591 24056 raft_consensus.cc:2454] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 1.
I20260620 14:53:40.633769 24197 raft_consensus.cc:2454] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c85f8031a4040038b5eb20dbd2bca6e in term 1.
I20260620 14:53:40.634105 24269 leader_election.cc:302] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c85f8031a4040038b5eb20dbd2bca6e, 2f539a55a32a4894b6e4ed891ae58622; no voters:
I20260620 14:53:40.634459 24413 raft_consensus.cc:2781] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:40.634634 24413 raft_consensus.cc:686] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 LEADER]: Becoming Leader. State: Replica: 0c85f8031a4040038b5eb20dbd2bca6e, State: Running, Role: LEADER
I20260620 14:53:40.635349 24413 consensus_queue.cc:237] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } }
I20260620 14:53:40.638057 23900 catalog_manager.cc:5495] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195). New cstate: current_term: 1 leader_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c85f8031a4040038b5eb20dbd2bca6e" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43543 } health_report { overall_health: HEALTHY } } }
I20260620 14:53:40.663249 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.656455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.21.159.195:43543)
0620 14:53:40.657119 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.657123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.657142 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:53:40.657198 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.657201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.657262 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.657348 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:40.658267 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.659127 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.661143 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.661671 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.661818 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.662367 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:40.662493 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:40.662569 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.662735 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.662802 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":383,"threads_started":1}
I20260620 14:53:40.674317 24056 raft_consensus.cc:1261] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Refusing update from remote peer 0c85f8031a4040038b5eb20dbd2bca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:40.675004 24197 raft_consensus.cc:1261] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Refusing update from remote peer 0c85f8031a4040038b5eb20dbd2bca6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:40.675652 24495 consensus_queue.cc:1035] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539a55a32a4894b6e4ed891ae58622" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:40.676203 24413 consensus_queue.cc:1035] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ceadd02f0c419ea39d1e4201750b6a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:53:40.711225 24483 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:53:42.302559 23900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:42.302810 23900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:42.307251 23900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed0638fb874450285b0a0116ec11e25: Sending DeleteTablet for 3 replicas of tablet 59c05bcc81fb4658a707af4c050a3a00
I20260620 14:53:42.308247 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c05bcc81fb4658a707af4c050a3a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:42 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:50100
I20260620 14:53:42.308326 24177 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c05bcc81fb4658a707af4c050a3a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:42 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:53718
I20260620 14:53:42.308449 24316 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c05bcc81fb4658a707af4c050a3a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:42 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54048
I20260620 14:53:42.309142 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23972
I20260620 14:53:42.323117 24562 tablet_replica.cc:330] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: stopping tablet replica
I20260620 14:53:42.323442 24562 raft_consensus.cc:2229] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:42.323902 24562 raft_consensus.cc:2258] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:42.324564 24561 tablet_replica.cc:330] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: stopping tablet replica
I20260620 14:53:42.324792 24561 raft_consensus.cc:2229] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:42.325034 24561 raft_consensus.cc:2258] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:42.325294 24562 ts_tablet_manager.cc:1905] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:42.325999 24561 ts_tablet_manager.cc:1905] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:42.332433 24562 ts_tablet_manager.cc:1918] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:53:42.332628 24562 log.cc:1196] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/59c05bcc81fb4658a707af4c050a3a00
I20260620 14:53:42.332994 24561 ts_tablet_manager.cc:1918] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:53:42.333076 24562 ts_tablet_manager.cc:1939] T 59c05bcc81fb4658a707af4c050a3a00 P 77ceadd02f0c419ea39d1e4201750b6a: Deleting consensus metadata
I20260620 14:53:42.333140 24561 log.cc:1196] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/59c05bcc81fb4658a707af4c050a3a00
I20260620 14:53:42.333554 24561 ts_tablet_manager.cc:1939] T 59c05bcc81fb4658a707af4c050a3a00 P 2f539a55a32a4894b6e4ed891ae58622: Deleting consensus metadata
I20260620 14:53:42.334364 23887 catalog_manager.cc:4841] TS 77ceadd02f0c419ea39d1e4201750b6a (127.21.159.194:45945): tablet 59c05bcc81fb4658a707af4c050a3a00 (table test-table [id=a73bd9c8306e41ecac80f029c3137a99]) successfully deleted
I20260620 14:53:42.335294 23887 catalog_manager.cc:4841] TS 2f539a55a32a4894b6e4ed891ae58622 (127.21.159.193:36747): tablet 59c05bcc81fb4658a707af4c050a3a00 (table test-table [id=a73bd9c8306e41ecac80f029c3137a99]) successfully deleted
I20260620 14:53:42.335566 24563 tablet_replica.cc:330] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: stopping tablet replica
I20260620 14:53:42.335804 24563 raft_consensus.cc:2229] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:42.336210 24563 raft_consensus.cc:2258] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:42.337283 24563 ts_tablet_manager.cc:1905] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:42.343606 24563 ts_tablet_manager.cc:1918] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:53:42.343811 24563 log.cc:1196] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/59c05bcc81fb4658a707af4c050a3a00
I20260620 14:53:42.344485 24563 ts_tablet_manager.cc:1939] T 59c05bcc81fb4658a707af4c050a3a00 P 0c85f8031a4040038b5eb20dbd2bca6e: Deleting consensus metadata
Jun 20 14:53:42 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:42.345856 23888 catalog_manager.cc:4841] TS 0c85f8031a4040038b5eb20dbd2bca6e (127.21.159.195:43543): tablet 59c05bcc81fb4658a707af4c050a3a00 (table test-table [id=a73bd9c8306e41ecac80f029c3137a99]) successfully deleted
I20260620 14:53:42.353900 24564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.313381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.21.159.193:36747)
0620 14:53:42.335041 (+ 21660us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:42.335046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:42.335099 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:53:42.335173 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:42.335179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:42.335274 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:42.335385 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:42.337214 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.338026 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.339895 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.340572 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.340707 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:42.346479 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:42.346497 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:42.346515 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:42.346562 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:42.350401 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.351638 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.351654 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.351657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.351829 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.352343 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.352348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.352351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.352794 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:42.353024 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:42.353370 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:53:42.353547 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21525,"thread_start_us":21459,"threads_started":1}
W20260620 14:53:42.354590 24103 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50118
W20260620 14:53:42.355314 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23972: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:42.355551 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 23972
I20260620 14:53:42.378357 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24112
Jun 20 14:53:42 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:53:42.396872 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.379273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.21.159.194:45945)
0620 14:53:42.379789 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:42.379793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:42.379820 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:53:42.379876 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:42.379879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:42.379950 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:42.380077 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:42.381384 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.382119 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.383550 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.384000 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.384153 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:42.388913 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:42.388930 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:42.388948 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:42.388992 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:42.392907 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.394133 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.394143 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.394145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.394323 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.394759 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.394765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.394768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.395209 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:42.395537 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:42.396287 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 14:53:42.396535 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":251,"threads_started":1}
W20260620 14:53:42.397625 24242 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53752
W20260620 14:53:42.398391 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24112: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:42.398581 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 24112
I20260620 14:53:42.425216 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24251
Jun 20 14:53:42 dist-test-slave-3mqx krb5kdc[23856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:53:42.442869 24567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.426040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.21.159.195:43543)
0620 14:53:42.426670 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:42.426675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:42.426693 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:53:42.426775 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:42.426778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:42.426845 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:42.426981 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:42.428407 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.429135 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.430431 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.430882 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.431038 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:42.436100 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:42.436117 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:42.436134 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:42.436177 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:42.439590 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.440477 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.440490 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.440495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.440694 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.441282 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.441290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.441294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.441795 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:42.442111 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:42.442388 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:53:42.442585 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":367,"threads_started":1}
W20260620 14:53:42.443581 24382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54074
W20260620 14:53:42.444294 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24251: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:42.444491 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 24251
I20260620 14:53:42.466856 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 23869
I20260620 14:53:42.468248 23941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:42.585139 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 23869
2026-06-20T14:53:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8679 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:42 dist-test-slave-3mqx krb5kdc[24571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:42 dist-test-slave-3mqx krb5kdc[24571](info): set up 2 sockets
Jun 20 14:53:42 dist-test-slave-3mqx krb5kdc[24571](info): commencing operation
krb5kdc: starting...
W20260620 14:53:43.667446 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:43 dist-test-slave-3mqx krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:43.870378 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42517
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:44.205436 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:44.205757 24584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:44.205910 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:44.213425 24584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:44.213557 24584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:44.213640 24584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:44.213723 24584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:44.213783 24584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:44.224123 24584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:44.225549 24584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:44.227007 24584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:44.240093 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:44.240176 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:44.240217 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:44.241194 24584 server_base.cc:1034] running on GCE node
I20260620 14:53:44.242064 24584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:44.243861 24584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:44.245172 24584 hybrid_clock.cc:648] HybridClock initialized: now 1781967224245129 us; error 59 us; skew 500 ppm
I20260620 14:53:44.245635 24584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:44 dist-test-slave-3mqx krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:44.249269 24584 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:44.250939 24584 webserver.cc:460] Webserver started at http://127.21.159.254:38405/ using document root <none> and password file <none>
I20260620 14:53:44.251709 24584 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:44.251812 24584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:44.252095 24584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:44.255172 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "3c64167f50ac45b9b8278e0a7b1d746a"
format_stamp: "Formatted at 2026-06-20 14:53:44 on dist-test-slave-3mqx"
server_key: "708c8be63026a5fb87e83f399679717e"
server_key_iv: "2aa0881679506369f11bf8ae26884ce8"
server_key_version: "encryptionkey@0"
I20260620 14:53:44.255936 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "3c64167f50ac45b9b8278e0a7b1d746a"
format_stamp: "Formatted at 2026-06-20 14:53:44 on dist-test-slave-3mqx"
server_key: "708c8be63026a5fb87e83f399679717e"
server_key_iv: "2aa0881679506369f11bf8ae26884ce8"
server_key_version: "encryptionkey@0"
I20260620 14:53:44.261335 24584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:53:44.264935 24599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:44.266395 24584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:53:44.266590 24584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "3c64167f50ac45b9b8278e0a7b1d746a"
format_stamp: "Formatted at 2026-06-20 14:53:44 on dist-test-slave-3mqx"
server_key: "708c8be63026a5fb87e83f399679717e"
server_key_iv: "2aa0881679506369f11bf8ae26884ce8"
server_key_version: "encryptionkey@0"
I20260620 14:53:44.266812 24584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:44.284189 24584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:44.288601 24584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:44.288913 24584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:44.308409 24584 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:42517
I20260620 14:53:44.308460 24650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:42517 every 8 connection(s)
I20260620 14:53:44.309870 24584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:44.310952 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 24584
I20260620 14:53:44.311296 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:44.312074 22143 external_mini_cluster.cc:1428] Setting key 5aa6a1cc1a0c8fd1adc21513bc535b54
I20260620 14:53:44.314941 24651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:53:44 dist-test-slave-3mqx krb5kdc[24571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:44.328920 24651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a: Bootstrap starting.
I20260620 14:53:44.332401 24651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:44.333472 24651 log.cc:824] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:44.336537 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:44.314181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.21.159.254:42517)
0620 14:53:44.315029 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:44.315047 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:44.315132 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:53:44.316822 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:44.316846 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:44.317257 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:44.317617 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:44.319494 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:44.320921 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:44.322772 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:44.323675 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:44.323830 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:44.328774 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:44.328783 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:44.328799 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:44.328843 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:44.332250 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:44.333088 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:44.333095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:44.333102 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:44.333280 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:44.333731 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:44.333735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:44.333737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:44.334300 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:44.334630 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:44.335101 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 14:53:44.335536 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":385,"threads_started":1}
I20260620 14:53:44.336556 24651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a: No bootstrap required, opened a new log
I20260620 14:53:44.345695 24651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42517 } }
I20260620 14:53:44.346138 24651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:44.346257 24651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c64167f50ac45b9b8278e0a7b1d746a, State: Initialized, Role: FOLLOWER
I20260620 14:53:44.346840 24651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42517 } }
I20260620 14:53:44.347117 24651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:44.347266 24651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:44.347453 24651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:44.349617 24651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42517 } }
I20260620 14:53:44.350149 24651 leader_election.cc:302] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c64167f50ac45b9b8278e0a7b1d746a; no voters:
I20260620 14:53:44.350728 24651 leader_election.cc:288] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:44.350891 24656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:44.352371 24656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [term 1 LEADER]: Becoming Leader. State: Replica: 3c64167f50ac45b9b8278e0a7b1d746a, State: Running, Role: LEADER
I20260620 14:53:44.352869 24656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42517 } }
I20260620 14:53:44.353044 24651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:44.355906 24658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c64167f50ac45b9b8278e0a7b1d746a. Latest consensus state: current_term: 1 leader_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42517 } } }
I20260620 14:53:44.356278 24658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:44.356002 24657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c64167f50ac45b9b8278e0a7b1d746a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42517 } } }
I20260620 14:53:44.356385 24657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:44.357986 24663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:44.363413 24663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:44.373795 24663 catalog_manager.cc:1347] Generated new cluster ID: a62008fa9f5a410e9728babdc69a57e1
I20260620 14:53:44.374001 24663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:44.392717 24663 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:44.394635 24663 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:44.408274 24663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c64167f50ac45b9b8278e0a7b1d746a: Generated new TSK 0
I20260620 14:53:44.409489 24663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:44.457178 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42517
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:44.818975 24677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:44.819377 24677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:44.819543 24677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:44.827379 24677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:44.827505 24677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:44.827683 24677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:44.838104 24677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:44.839577 24677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:44.841040 24677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:44.855788 24685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:44.855904 24683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:44.855808 24682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:44.856662 24677 server_base.cc:1034] running on GCE node
I20260620 14:53:44.857404 24677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:44.858476 24677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:44.859701 24677 hybrid_clock.cc:648] HybridClock initialized: now 1781967224859608 us; error 67 us; skew 500 ppm
I20260620 14:53:44.860060 24677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:44 dist-test-slave-3mqx krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:44.863512 24677 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:44.865235 24677 webserver.cc:460] Webserver started at http://127.21.159.193:40893/ using document root <none> and password file <none>
I20260620 14:53:44.866058 24677 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:44.866217 24677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:44.866559 24677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:44.869962 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "837d1696923b4553adddd7c71269286c"
format_stamp: "Formatted at 2026-06-20 14:53:44 on dist-test-slave-3mqx"
server_key: "e897d12f57cb618688d5535b2a959f29"
server_key_iv: "dd42794b9fe5c335fd3c50ede806ddee"
server_key_version: "encryptionkey@0"
I20260620 14:53:44.870870 24677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "837d1696923b4553adddd7c71269286c"
format_stamp: "Formatted at 2026-06-20 14:53:44 on dist-test-slave-3mqx"
server_key: "e897d12f57cb618688d5535b2a959f29"
server_key_iv: "dd42794b9fe5c335fd3c50ede806ddee"
server_key_version: "encryptionkey@0"
I20260620 14:53:44.876602 24677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 14:53:44.880789 24692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:44.882336 24677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:53:44.882576 24677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "837d1696923b4553adddd7c71269286c"
format_stamp: "Formatted at 2026-06-20 14:53:44 on dist-test-slave-3mqx"
server_key: "e897d12f57cb618688d5535b2a959f29"
server_key_iv: "dd42794b9fe5c335fd3c50ede806ddee"
server_key_version: "encryptionkey@0"
I20260620 14:53:44.882809 24677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:44.894165 24677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:44.898746 24677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:44.899106 24677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:44.900110 24677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:44.901510 24677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:44.901613 24677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:44.901760 24677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:44.901832 24677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:44.935861 24677 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:32793
I20260620 14:53:44.935913 24804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:32793 every 8 connection(s)
I20260620 14:53:44.937247 24677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:44.939311 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 24677
I20260620 14:53:44.939539 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:44.940395 22143 external_mini_cluster.cc:1428] Setting key c2bdfb057de14baca2ff797100bfb503
Jun 20 14:53:44 dist-test-slave-3mqx krb5kdc[24571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967224, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:53:44.961292 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:44.940205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:42755 (local address 127.21.159.254:42517)
0620 14:53:44.940683 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:44.940687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:44.942015 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 14:53:44.944003 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:44.944013 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:44.944107 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:44.944233 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:44.945886 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:44.947159 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:44.949950 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:44.950373 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:44.950516 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:44.955176 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:44.955181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:44.955185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:44.955224 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:44.957848 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:44.958728 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:44.958732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:44.958735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:44.958854 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:44.959458 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:44.959463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:44.959467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:44.959784 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:44.959901 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:44.960829 (+ 928us) server_negotiation.cc:299] Negotiation successful
0620 14:53:44.961020 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":284,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:53:44.962456 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:44.940950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42517 (local address 127.21.159.193:42755)
0620 14:53:44.941826 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:44.941879 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:44.943698 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:44.944480 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:44.944488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:44.945069 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:44.945684 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:44.945703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:44.947345 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:44.947349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:44.949753 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:44.949774 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:44.950638 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:44.950648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:44.951020 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:44.951859 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:44.951895 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:44.954867 (+ 2972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:44.958036 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:44.958047 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:44.958069 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:44.958567 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:44.958982 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:44.958987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:44.958991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:44.959275 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:44.959926 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:44.959933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:44.960361 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 14:53:44.960709 (+ 348us) client_negotiation.cc:240] Negotiation successful
0620 14:53:44.961167 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":314,"threads_started":1}
I20260620 14:53:44.964385 24805 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42517
I20260620 14:53:44.964764 24805 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:44.965734 24805 heartbeater.cc:511] Master 127.21.159.254:42517 requested a full tablet report, sending...
I20260620 14:53:44.968523 24616 ts_manager.cc:194] Registered new tserver with Master: 837d1696923b4553adddd7c71269286c (127.21.159.193:32793)
I20260620 14:53:44.970255 24616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:42755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:44.991881 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42517
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:45.328961 24811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:45.329324 24811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:45.329504 24811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:45.336877 24811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:45.337057 24811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:45.337248 24811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:45.347276 24811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:45.348716 24811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:45.350162 24811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:45.364671 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:45.364614 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:45.364670 24819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:45.365613 24811 server_base.cc:1034] running on GCE node
I20260620 14:53:45.366255 24811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:45.367282 24811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:45.368527 24811 hybrid_clock.cc:648] HybridClock initialized: now 1781967225368446 us; error 64 us; skew 500 ppm
I20260620 14:53:45.368990 24811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:45 dist-test-slave-3mqx krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967225, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:45.372581 24811 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:45.374197 24811 webserver.cc:460] Webserver started at http://127.21.159.194:42847/ using document root <none> and password file <none>
I20260620 14:53:45.374967 24811 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:45.375160 24811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:45.375487 24811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:45.378723 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "db593ebe5e4643a38dc4aa8cfd68fd06"
format_stamp: "Formatted at 2026-06-20 14:53:45 on dist-test-slave-3mqx"
server_key: "2647c741f3e26cba65b32b4fa08b514f"
server_key_iv: "6e83d3a0d7d1fbb7bb21f9f29975312a"
server_key_version: "encryptionkey@0"
I20260620 14:53:45.379735 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "db593ebe5e4643a38dc4aa8cfd68fd06"
format_stamp: "Formatted at 2026-06-20 14:53:45 on dist-test-slave-3mqx"
server_key: "2647c741f3e26cba65b32b4fa08b514f"
server_key_iv: "6e83d3a0d7d1fbb7bb21f9f29975312a"
server_key_version: "encryptionkey@0"
I20260620 14:53:45.385383 24811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 14:53:45.389458 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:45.391215 24811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:45.391430 24811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "db593ebe5e4643a38dc4aa8cfd68fd06"
format_stamp: "Formatted at 2026-06-20 14:53:45 on dist-test-slave-3mqx"
server_key: "2647c741f3e26cba65b32b4fa08b514f"
server_key_iv: "6e83d3a0d7d1fbb7bb21f9f29975312a"
server_key_version: "encryptionkey@0"
I20260620 14:53:45.391685 24811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:45.407788 24811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:45.411957 24811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:45.412256 24811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:45.413242 24811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:45.414631 24811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:45.414749 24811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:45.414898 24811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:45.414975 24811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:45.448511 24811 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:36205
I20260620 14:53:45.448557 24938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:36205 every 8 connection(s)
I20260620 14:53:45.449831 24811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:45.451673 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 24811
I20260620 14:53:45.451864 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:45.452288 22143 external_mini_cluster.cc:1428] Setting key 0c6ded6bd9c846904f9901658aa17b65
Jun 20 14:53:45 dist-test-slave-3mqx krb5kdc[24571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967225, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:53:45.471292 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:45.452576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:36951 (local address 127.21.159.254:42517)
0620 14:53:45.452793 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:45.452799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:45.454264 (+ 1465us) server_negotiation.cc:407] Connection header received
0620 14:53:45.456226 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:45.456232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:45.456317 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:45.456409 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:45.458000 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.458713 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.460467 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.460928 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.461066 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:45.465114 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:45.465119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:45.465123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:45.465158 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:45.467992 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:45.468817 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:45.468821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:45.468823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:45.468946 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:45.469433 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:45.469442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:45.469444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:45.469704 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:45.469861 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:45.470672 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 14:53:45.470905 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:53:45.472397 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:45.453227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42517 (local address 127.21.159.194:36951)
0620 14:53:45.454090 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:45.454143 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:45.455950 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:45.456659 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:45.456667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:45.457256 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:45.457828 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:45.457845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.458876 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.458888 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:45.460326 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:45.460339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.461163 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.461170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:45.461436 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:45.462174 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:45.462197 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:45.464822 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:45.468166 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:45.468176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:45.468198 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:45.468647 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:45.469063 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:45.469070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:45.469075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:45.469305 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:45.469835 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:45.469844 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:45.470307 (+ 463us) client_negotiation.cc:769] Sending connection context
0620 14:53:45.470577 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 14:53:45.471188 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":341,"threads_started":1}
I20260620 14:53:45.473954 24939 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42517
I20260620 14:53:45.474313 24939 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:45.475289 24939 heartbeater.cc:511] Master 127.21.159.254:42517 requested a full tablet report, sending...
I20260620 14:53:45.477001 24616 ts_manager.cc:194] Registered new tserver with Master: db593ebe5e4643a38dc4aa8cfd68fd06 (127.21.159.194:36205)
I20260620 14:53:45.477843 24616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:36951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:45.498210 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42517
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:45.835394 24944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:45.835826 24944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:45.836021 24944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:45.843555 24944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:45.843686 24944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:45.843905 24944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:45.854094 24944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:45.855578 24944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:45.857070 24944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:45.871510 24950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:45.871546 24952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:45.871878 24949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:45.872560 24944 server_base.cc:1034] running on GCE node
I20260620 14:53:45.873235 24944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:45.874269 24944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:45.875509 24944 hybrid_clock.cc:648] HybridClock initialized: now 1781967225875451 us; error 42 us; skew 500 ppm
I20260620 14:53:45.875913 24944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:45 dist-test-slave-3mqx krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967225, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:45.879549 24944 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:45.881369 24944 webserver.cc:460] Webserver started at http://127.21.159.195:35989/ using document root <none> and password file <none>
I20260620 14:53:45.882225 24944 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:45.882398 24944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:45.882761 24944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:45.886327 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "c7bc9a26ea88481ca0fd24657c1ca20e"
format_stamp: "Formatted at 2026-06-20 14:53:45 on dist-test-slave-3mqx"
server_key: "7aa6e0b5b88aab5dfdbb7c4ba401e665"
server_key_iv: "58731e723d0d4549120a4d856176420c"
server_key_version: "encryptionkey@0"
I20260620 14:53:45.887389 24944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "c7bc9a26ea88481ca0fd24657c1ca20e"
format_stamp: "Formatted at 2026-06-20 14:53:45 on dist-test-slave-3mqx"
server_key: "7aa6e0b5b88aab5dfdbb7c4ba401e665"
server_key_iv: "58731e723d0d4549120a4d856176420c"
server_key_version: "encryptionkey@0"
I20260620 14:53:45.893504 24944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:53:45.897838 24959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:45.899407 24944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:45.899617 24944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "c7bc9a26ea88481ca0fd24657c1ca20e"
format_stamp: "Formatted at 2026-06-20 14:53:45 on dist-test-slave-3mqx"
server_key: "7aa6e0b5b88aab5dfdbb7c4ba401e665"
server_key_iv: "58731e723d0d4549120a4d856176420c"
server_key_version: "encryptionkey@0"
I20260620 14:53:45.899852 24944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:45.930960 24944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:45.935554 24944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:45.935865 24944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:45.936936 24944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:45.938494 24944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:45.938604 24944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:45.938762 24944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:45.938843 24944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:45.973615 24944 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:43691
I20260620 14:53:45.973621 25071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:43691 every 8 connection(s)
I20260620 14:53:45.974309 24805 heartbeater.cc:503] Master 127.21.159.254:42517 was elected leader, sending a full tablet report...
I20260620 14:53:45.975277 24944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:45.978746 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 24944
I20260620 14:53:45.978953 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:45.979607 22143 external_mini_cluster.cc:1428] Setting key 508cca9f92a08177d79156618e2bcc4f
Jun 20 14:53:45 dist-test-slave-3mqx krb5kdc[24571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967225, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:45.998498 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:45.978029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:59841 (local address 127.21.159.254:42517)
0620 14:53:45.978487 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:45.978492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:45.979943 (+ 1451us) server_negotiation.cc:407] Connection header received
0620 14:53:45.982172 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:45.982177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:45.982263 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:45.982383 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:45.984437 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.985328 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.987416 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.987911 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.988020 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:45.991855 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:45.991860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:45.991864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:45.991900 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:45.995123 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:45.996073 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:45.996077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:45.996079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:45.996295 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:45.996741 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:45.996744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:45.996746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:45.997074 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:45.997186 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:45.998031 (+ 845us) server_negotiation.cc:299] Negotiation successful
0620 14:53:45.998221 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":243,"threads_started":1}
I20260620 14:53:45.999604 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:45.978829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42517 (local address 127.21.159.195:59841)
0620 14:53:45.979768 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:45.979829 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:45.981898 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:45.982740 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:45.982750 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:45.983541 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:45.984207 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:45.984228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.985524 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.985528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:45.987265 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:45.987279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:45.988131 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:45.988136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:45.988356 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:45.989061 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:45.989090 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:45.991576 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:45.995304 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:45.995315 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:45.995335 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:45.995905 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:45.996410 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:45.996420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:45.996422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:45.996620 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:45.997208 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:45.997215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:45.997619 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 14:53:45.997874 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 14:53:45.998291 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":342,"threads_started":1}
I20260620 14:53:46.001215 25072 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42517
I20260620 14:53:46.001579 25072 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:46.002348 25072 heartbeater.cc:511] Master 127.21.159.254:42517 requested a full tablet report, sending...
I20260620 14:53:46.003932 24616 ts_manager.cc:194] Registered new tserver with Master: c7bc9a26ea88481ca0fd24657c1ca20e (127.21.159.195:43691)
I20260620 14:53:46.004859 24616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:59841
I20260620 14:53:46.008265 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:53:46.039983 25074 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:46.037195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.21.159.254:42517)
0620 14:53:46.037423 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:46.037431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:46.037887 (+ 456us) server_negotiation.cc:407] Connection header received
0620 14:53:46.038038 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:46.038042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:46.038108 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:46.038230 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:46.039639 (+ 1409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46480: BlockingRecv error: recv got EOF from 127.0.0.1:46480 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:53:46.043622 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24677
W20260620 14:53:46.047551 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24677: Not authorized: Client connection negotiation failed: client connection to 127.21.159.193:32793: server requires authentication, but client does not have Kerberos credentials available
I20260620 14:53:46.047760 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 24677
I20260620 14:53:46.064345 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24811
W20260620 14:53:46.067070 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24811: Not authorized: Client connection negotiation failed: client connection to 127.21.159.194:36205: server requires authentication, but client does not have Kerberos credentials available
I20260620 14:53:46.067245 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 24811
I20260620 14:53:46.082909 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24944
W20260620 14:53:46.086112 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24944: Not authorized: Client connection negotiation failed: client connection to 127.21.159.195:43691: server requires authentication, but client does not have Kerberos credentials available
I20260620 14:53:46.086273 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 24944
I20260620 14:53:46.101981 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 24584
I20260620 14:53:46.103268 24646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:46.205976 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 24584
2026-06-20T14:53:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3617 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:46 dist-test-slave-3mqx krb5kdc[25090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:46 dist-test-slave-3mqx krb5kdc[25090](info): set up 2 sockets
Jun 20 14:53:46 dist-test-slave-3mqx krb5kdc[25090](info): commencing operation
krb5kdc: starting...
W20260620 14:53:47.351488 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:47 dist-test-slave-3mqx krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:47.558883 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35831
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:35831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:47.912109 25103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:47.912458 25103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:47.912638 25103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:47.921088 25103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:47.921231 25103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:47.921325 25103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:47.921430 25103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:47.921528 25103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:47.932927 25103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:35831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:47.934473 25103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:47.936128 25103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:47.950031 25109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:47.950129 25111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:47.950073 25108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:47.951509 25103 server_base.cc:1034] running on GCE node
I20260620 14:53:47.952476 25103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:47.954139 25103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:47.955404 25103 hybrid_clock.cc:648] HybridClock initialized: now 1781967227955354 us; error 52 us; skew 500 ppm
I20260620 14:53:47.955842 25103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:47 dist-test-slave-3mqx krb5kdc[25090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967227, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:47.959574 25103 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:47.961369 25103 webserver.cc:460] Webserver started at http://127.21.159.254:37185/ using document root <none> and password file <none>
I20260620 14:53:47.962137 25103 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:47.962292 25103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:47.962643 25103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:47.965945 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "bddaa3db0b47493baa86bb871dd7c6d7"
format_stamp: "Formatted at 2026-06-20 14:53:47 on dist-test-slave-3mqx"
server_key: "bdac9b334cb20457bd54aa4eaf9c7fbe"
server_key_iv: "b5f48b9f470c6f5edb4e2ddc66606334"
server_key_version: "encryptionkey@0"
I20260620 14:53:47.966882 25103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "bddaa3db0b47493baa86bb871dd7c6d7"
format_stamp: "Formatted at 2026-06-20 14:53:47 on dist-test-slave-3mqx"
server_key: "bdac9b334cb20457bd54aa4eaf9c7fbe"
server_key_iv: "b5f48b9f470c6f5edb4e2ddc66606334"
server_key_version: "encryptionkey@0"
I20260620 14:53:47.973107 25103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:53:47.977113 25118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:47.978675 25103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:53:47.978895 25103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "bddaa3db0b47493baa86bb871dd7c6d7"
format_stamp: "Formatted at 2026-06-20 14:53:47 on dist-test-slave-3mqx"
server_key: "bdac9b334cb20457bd54aa4eaf9c7fbe"
server_key_iv: "b5f48b9f470c6f5edb4e2ddc66606334"
server_key_version: "encryptionkey@0"
I20260620 14:53:47.979146 25103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:48.010278 25103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:48.015623 25103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:48.016018 25103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:48.037153 25103 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:35831
I20260620 14:53:48.037181 25169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:35831 every 8 connection(s)
I20260620 14:53:48.039218 25103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:48.040951 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25103
I20260620 14:53:48.041250 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:48.042068 22143 external_mini_cluster.cc:1428] Setting key 9786b11966982e7d977e806485b65594
I20260620 14:53:48.044696 25170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:48.059333 25170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7: Bootstrap starting.
Jun 20 14:53:48 dist-test-slave-3mqx krb5kdc[25090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:48.063364 25170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:48.064585 25170 log.cc:824] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:48.067803 25170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7: No bootstrap required, opened a new log
I20260620 14:53:48.070338 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:48.044274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.21.159.254:35831)
0620 14:53:48.045283 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:48.045296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:48.045351 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:53:48.047248 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:48.047285 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:48.047784 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:48.048217 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:48.050414 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:48.052361 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:48.054354 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:48.055836 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:48.056109 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:48.061169 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:48.061178 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:48.061205 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:48.061261 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:48.065129 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:48.066080 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:48.066088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:48.066096 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:48.066298 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:48.066786 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:48.066797 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:48.066799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:48.067468 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:48.067797 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:48.068370 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 14:53:48.068882 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":520,"threads_started":1}
I20260620 14:53:48.075220 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:48.076872 25170 raft_consensus.cc:348] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35831 } }
I20260620 14:53:48.077351 25170 raft_consensus.cc:374] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:48.077483 25170 raft_consensus.cc:729] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bddaa3db0b47493baa86bb871dd7c6d7, State: Initialized, Role: FOLLOWER
I20260620 14:53:48.078105 25170 consensus_queue.cc:260] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35831 } }
I20260620 14:53:48.078343 25170 raft_consensus.cc:388] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:48.078472 25170 raft_consensus.cc:482] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:48.078680 25170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:48.081007 25170 raft_consensus.cc:504] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35831 } }
I20260620 14:53:48.081627 25170 leader_election.cc:302] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bddaa3db0b47493baa86bb871dd7c6d7; no voters:
I20260620 14:53:48.082348 25170 leader_election.cc:288] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:48.082455 25176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:48.085189 25170 sys_catalog.cc:564] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:48.090885 25176 raft_consensus.cc:686] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [term 1 LEADER]: Becoming Leader. State: Replica: bddaa3db0b47493baa86bb871dd7c6d7, State: Running, Role: LEADER
I20260620 14:53:48.091955 25176 consensus_queue.cc:237] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35831 } }
I20260620 14:53:48.096580 25178 sys_catalog.cc:455] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bddaa3db0b47493baa86bb871dd7c6d7. Latest consensus state: current_term: 1 leader_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35831 } } }
I20260620 14:53:48.097136 25178 sys_catalog.cc:458] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:48.096622 25177 sys_catalog.cc:455] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bddaa3db0b47493baa86bb871dd7c6d7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35831 } } }
I20260620 14:53:48.100479 25177 sys_catalog.cc:458] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7 [sys.catalog]: This master's current role is: LEADER
W20260620 14:53:48.106950 25190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:53:48.107223 25190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:53:48.107379 25191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:48.109817 25191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:48.121213 25191 catalog_manager.cc:1347] Generated new cluster ID: de1c1974734c40dfbb47efb3c5ed329f
I20260620 14:53:48.121356 25191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:48.128266 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:48.117411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.21.159.254:35831)
0620 14:53:48.117617 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:48.117625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:48.118227 (+ 602us) server_negotiation.cc:407] Connection header received
0620 14:53:48.118501 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:48.118509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:48.118611 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:48.118729 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:48.120218 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:48.121587 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:48.123670 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:48.124529 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:48.124700 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:48.126020 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:48.126028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:48.126033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:53:48.127038 (+ 1005us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:48.127140 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:53:48.127174 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:48.127307 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:48.127489 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 14:53:48.127617 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:53:48.131740 25191 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:48.133901 25191 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:48.146183 25191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bddaa3db0b47493baa86bb871dd7c6d7: Generated new TSK 0
I20260620 14:53:48.147302 25191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:48.161898 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25103
I20260620 14:53:48.163390 25165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:48.254029 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25103
2026-06-20T14:53:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2040 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:48 dist-test-slave-3mqx krb5kdc[25204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:48 dist-test-slave-3mqx krb5kdc[25204](info): set up 2 sockets
Jun 20 14:53:48 dist-test-slave-3mqx krb5kdc[25204](info): commencing operation
krb5kdc: starting...
W20260620 14:53:49.387581 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:49 dist-test-slave-3mqx krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:49.595686 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36335
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:36335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:49.940515 25217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:49.940847 25217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:49.940999 25217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:49.948720 25217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:49.948859 25217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:49.948937 25217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:49.949026 25217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:49.949110 25217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:49.960315 25217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:36335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:49.961781 25217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:49.963225 25217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:49.975574 25223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:49.975878 25222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:49.976166 25225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:49.976759 25217 server_base.cc:1034] running on GCE node
I20260620 14:53:49.977648 25217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:49.979384 25217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:49.980630 25217 hybrid_clock.cc:648] HybridClock initialized: now 1781967229980590 us; error 54 us; skew 500 ppm
I20260620 14:53:49.980994 25217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:49 dist-test-slave-3mqx krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967229, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:49.984498 25217 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:49.986258 25217 webserver.cc:460] Webserver started at http://127.21.159.254:36855/ using document root <none> and password file <none>
I20260620 14:53:49.986981 25217 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:49.987149 25217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:49.987474 25217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:49.990787 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "654a2c3086d34de7aa9f189f721c8040"
format_stamp: "Formatted at 2026-06-20 14:53:49 on dist-test-slave-3mqx"
server_key: "a9c1fa97f41df52382f7ef095ce8a8de"
server_key_iv: "14e6e84b1a767da5860d7b67e8a6928c"
server_key_version: "encryptionkey@0"
I20260620 14:53:49.991688 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "654a2c3086d34de7aa9f189f721c8040"
format_stamp: "Formatted at 2026-06-20 14:53:49 on dist-test-slave-3mqx"
server_key: "a9c1fa97f41df52382f7ef095ce8a8de"
server_key_iv: "14e6e84b1a767da5860d7b67e8a6928c"
server_key_version: "encryptionkey@0"
I20260620 14:53:49.997546 25217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 14:53:50.001869 25232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:50.003705 25217 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:53:50.003993 25217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "654a2c3086d34de7aa9f189f721c8040"
format_stamp: "Formatted at 2026-06-20 14:53:49 on dist-test-slave-3mqx"
server_key: "a9c1fa97f41df52382f7ef095ce8a8de"
server_key_iv: "14e6e84b1a767da5860d7b67e8a6928c"
server_key_version: "encryptionkey@0"
I20260620 14:53:50.004225 25217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:50.029561 25217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:50.034633 25217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:50.035012 25217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:50.056850 25217 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:36335
I20260620 14:53:50.056864 25283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:36335 every 8 connection(s)
I20260620 14:53:50.058439 25217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:50.063817 25284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:50.067021 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25217
I20260620 14:53:50.067358 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:50.068041 22143 external_mini_cluster.cc:1428] Setting key 83ebd0bdde37df09a8ddc52376c282f4
I20260620 14:53:50.078049 25284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040: Bootstrap starting.
I20260620 14:53:50.082532 25284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:50.084208 25284 log.cc:824] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:53:50 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:50.088392 25284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040: No bootstrap required, opened a new log
I20260620 14:53:50.096652 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:50.070485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.21.159.254:36335)
0620 14:53:50.071434 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:50.071448 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:50.071494 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:53:50.073286 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:50.073312 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:50.073794 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:50.074163 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:50.076231 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:50.077880 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:50.079869 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:50.080726 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:50.080938 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:50.086352 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:50.086365 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:50.086389 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:50.086442 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:50.091001 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:50.091992 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:50.092008 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:50.092018 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:50.092263 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:50.092730 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:50.092735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:50.092740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:50.093601 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:50.093985 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:50.094425 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:53:50.094989 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":429,"threads_started":1}
I20260620 14:53:50.100514 25284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654a2c3086d34de7aa9f189f721c8040" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36335 } }
I20260620 14:53:50.100936 25284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:50.101017 25284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654a2c3086d34de7aa9f189f721c8040, State: Initialized, Role: FOLLOWER
I20260620 14:53:50.101563 25284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654a2c3086d34de7aa9f189f721c8040" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36335 } }
I20260620 14:53:50.101832 25284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:50.101986 25284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:50.102175 25284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:50.104984 25284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654a2c3086d34de7aa9f189f721c8040" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36335 } }
I20260620 14:53:50.105571 25284 leader_election.cc:302] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 654a2c3086d34de7aa9f189f721c8040; no voters:
I20260620 14:53:50.106323 25284 leader_election.cc:288] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:50.106479 25289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:50.108099 25289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [term 1 LEADER]: Becoming Leader. State: Replica: 654a2c3086d34de7aa9f189f721c8040, State: Running, Role: LEADER
I20260620 14:53:50.108911 25289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654a2c3086d34de7aa9f189f721c8040" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36335 } }
I20260620 14:53:50.109130 25284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:50.112696 25290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "654a2c3086d34de7aa9f189f721c8040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654a2c3086d34de7aa9f189f721c8040" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36335 } } }
I20260620 14:53:50.112517 25291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 654a2c3086d34de7aa9f189f721c8040. Latest consensus state: current_term: 1 leader_uuid: "654a2c3086d34de7aa9f189f721c8040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "654a2c3086d34de7aa9f189f721c8040" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36335 } } }
I20260620 14:53:50.113528 25290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:50.113601 25291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:50.116139 25297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:50.122182 25297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:50.132997 25297 catalog_manager.cc:1347] Generated new cluster ID: e157acc753c24a05b3fb8db5e7f94460
I20260620 14:53:50.133147 25297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:50.153383 25297 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:50.155611 25297 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:50.172178 25297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 654a2c3086d34de7aa9f189f721c8040: Generated new TSK 0
I20260620 14:53:50.173435 25297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:50.227988 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36335
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:50.613204 25310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:50.613531 25310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:50.613655 25310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:50.622517 25310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:50.622715 25310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:50.622860 25310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:50.633302 25310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:50.634698 25310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:50.636216 25310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:50.650743 25315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:50.650743 25316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:50.650786 25318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:50.651966 25310 server_base.cc:1034] running on GCE node
I20260620 14:53:50.652843 25310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:50.653978 25310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:50.655292 25310 hybrid_clock.cc:648] HybridClock initialized: now 1781967230655209 us; error 73 us; skew 500 ppm
I20260620 14:53:50.655697 25310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:50 dist-test-slave-3mqx krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967230, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:50.659794 25310 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:50.661589 25310 webserver.cc:460] Webserver started at http://127.21.159.193:40625/ using document root <none> and password file <none>
I20260620 14:53:50.662338 25310 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:50.662452 25310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:50.662721 25310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:50.666018 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "170c8c2fdadd47b6a56d5b3d31d48ee1"
format_stamp: "Formatted at 2026-06-20 14:53:50 on dist-test-slave-3mqx"
server_key: "ef0431f6e7201af8576be892ceb7c38a"
server_key_iv: "bee33aa5d43c4ec6b4dffbdd7925a593"
server_key_version: "encryptionkey@0"
I20260620 14:53:50.666923 25310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "170c8c2fdadd47b6a56d5b3d31d48ee1"
format_stamp: "Formatted at 2026-06-20 14:53:50 on dist-test-slave-3mqx"
server_key: "ef0431f6e7201af8576be892ceb7c38a"
server_key_iv: "bee33aa5d43c4ec6b4dffbdd7925a593"
server_key_version: "encryptionkey@0"
I20260620 14:53:50.673108 25310 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260620 14:53:50.677640 25325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:50.679482 25310 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:53:50.679718 25310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "170c8c2fdadd47b6a56d5b3d31d48ee1"
format_stamp: "Formatted at 2026-06-20 14:53:50 on dist-test-slave-3mqx"
server_key: "ef0431f6e7201af8576be892ceb7c38a"
server_key_iv: "bee33aa5d43c4ec6b4dffbdd7925a593"
server_key_version: "encryptionkey@0"
I20260620 14:53:50.679937 25310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:50.715013 25310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:50.720557 25310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:50.720829 25310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:50.721809 25310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:50.723340 25310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:50.723415 25310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:50.723511 25310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:50.723557 25310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:50.758816 25310 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:46495
I20260620 14:53:50.758915 25437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:46495 every 8 connection(s)
I20260620 14:53:50.760263 25310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:50.761750 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25310
I20260620 14:53:50.762022 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:50.762755 22143 external_mini_cluster.cc:1428] Setting key c52e1bdccd0a30d27d41c2b8e49de9a0
Jun 20 14:53:50 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967230, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:53:50.783716 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:50.763344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:40911 (local address 127.21.159.254:36335)
0620 14:53:50.763877 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:50.763884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:50.765104 (+ 1220us) server_negotiation.cc:407] Connection header received
0620 14:53:50.767034 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:50.767039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:50.767157 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:50.767291 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:50.768882 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:50.769613 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:50.771329 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:50.772252 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:50.772428 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:50.776244 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:50.776255 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:50.776263 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:50.776316 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:50.780011 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:50.780895 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:50.780902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:50.780906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:50.781042 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:50.781547 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:50.781557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:50.781560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:50.781868 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:50.782039 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:50.782994 (+ 955us) server_negotiation.cc:299] Negotiation successful
0620 14:53:50.783311 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":255,"threads_started":1}
I20260620 14:53:50.784793 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:50.764112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36335 (local address 127.21.159.193:40911)
0620 14:53:50.764940 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:50.764990 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:50.766736 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:50.767554 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:50.767564 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:50.768106 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:50.768699 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:50.768717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:50.769791 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:50.769795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:50.771154 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:50.771167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:50.772514 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:50.772519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:50.772753 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:50.773522 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:50.773551 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:50.775949 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:50.780231 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:50.780240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:50.780256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:50.780735 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:50.781190 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:50.781195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:50.781197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:50.781410 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:50.782013 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:50.782024 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:50.782432 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 14:53:50.782737 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 14:53:50.783272 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":353,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:53:50.786811 25438 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36335
I20260620 14:53:50.787286 25438 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:50.788076 25438 heartbeater.cc:511] Master 127.21.159.254:36335 requested a full tablet report, sending...
I20260620 14:53:50.790393 25249 ts_manager.cc:194] Registered new tserver with Master: 170c8c2fdadd47b6a56d5b3d31d48ee1 (127.21.159.193:46495)
I20260620 14:53:50.792006 25249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:40911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:50.811745 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36335
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:51.179177 25444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:51.179530 25444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:51.179735 25444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:51.187873 25444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:51.188004 25444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:51.188194 25444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:51.199544 25444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:51.201107 25444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:51.202628 25444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:51.217454 25450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:51.217520 25449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:51.217490 25452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:51.218425 25444 server_base.cc:1034] running on GCE node
I20260620 14:53:51.219229 25444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:51.220275 25444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:51.221522 25444 hybrid_clock.cc:648] HybridClock initialized: now 1781967231221463 us; error 48 us; skew 500 ppm
I20260620 14:53:51.221856 25444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:51.225368 25444 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:51.226966 25444 webserver.cc:460] Webserver started at http://127.21.159.194:41285/ using document root <none> and password file <none>
I20260620 14:53:51.227731 25444 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:51.227885 25444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:51.228199 25444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:51.231383 25444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "49c8bd61c3464c0a8b1c07b1f15e031e"
format_stamp: "Formatted at 2026-06-20 14:53:51 on dist-test-slave-3mqx"
server_key: "a02013e6bc9028d54a1bf4888fa5d69c"
server_key_iv: "7b7a9bf1488aae044b1e87bb387bb117"
server_key_version: "encryptionkey@0"
I20260620 14:53:51.232288 25444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "49c8bd61c3464c0a8b1c07b1f15e031e"
format_stamp: "Formatted at 2026-06-20 14:53:51 on dist-test-slave-3mqx"
server_key: "a02013e6bc9028d54a1bf4888fa5d69c"
server_key_iv: "7b7a9bf1488aae044b1e87bb387bb117"
server_key_version: "encryptionkey@0"
I20260620 14:53:51.237874 25444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:53:51.241652 25459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:51.243278 25444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:53:51.243456 25444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "49c8bd61c3464c0a8b1c07b1f15e031e"
format_stamp: "Formatted at 2026-06-20 14:53:51 on dist-test-slave-3mqx"
server_key: "a02013e6bc9028d54a1bf4888fa5d69c"
server_key_iv: "7b7a9bf1488aae044b1e87bb387bb117"
server_key_version: "encryptionkey@0"
I20260620 14:53:51.243670 25444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:51.276180 25444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:51.280735 25444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:51.281075 25444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:51.282194 25444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:51.283991 25444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:51.284117 25444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:51.284260 25444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:51.284332 25444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:51.319618 25444 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:43007
I20260620 14:53:51.319721 25571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:43007 every 8 connection(s)
I20260620 14:53:51.321091 25444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:51.324667 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25444
I20260620 14:53:51.324877 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:51.325436 22143 external_mini_cluster.cc:1428] Setting key 8a0a39cc96ba02ff6031dea2a58ffcb6
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:53:51.349373 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.324048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:54743 (local address 127.21.159.254:36335)
0620 14:53:51.324914 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:51.324921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:51.326016 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 14:53:51.328220 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:51.328229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:51.328329 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:51.328477 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:51.330114 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.331473 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.333572 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.334042 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.334150 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:51.342262 (+ 8112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:51.342267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:51.342271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:51.342308 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:51.345510 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.346479 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.346488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.346492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.346646 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.347245 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.347250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.347252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.347571 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:51.347749 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:51.348850 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0620 14:53:51.349087 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":226,"threads_started":1}
I20260620 14:53:51.351181 25575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.324865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36335 (local address 127.21.159.194:54743)
0620 14:53:51.325810 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:51.325866 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:51.327880 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:51.328744 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:51.328753 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:51.329308 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:51.329915 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:51.329934 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.331680 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.331684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:51.333381 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:51.333401 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.334291 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.334300 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:51.334653 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:51.338876 (+ 4223us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:51.338922 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:51.341964 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:51.345724 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:51.345737 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:51.345762 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:51.346311 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:51.346792 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:51.346798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:51.346802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:51.347029 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:51.347722 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:51.347732 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:51.348263 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 14:53:51.348644 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 14:53:51.349356 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":318,"threads_started":1}
I20260620 14:53:51.352928 25572 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36335
I20260620 14:53:51.353282 25572 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:51.354122 25572 heartbeater.cc:511] Master 127.21.159.254:36335 requested a full tablet report, sending...
I20260620 14:53:51.355835 25249 ts_manager.cc:194] Registered new tserver with Master: 49c8bd61c3464c0a8b1c07b1f15e031e (127.21.159.194:43007)
I20260620 14:53:51.357156 25249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:54743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:51.375329 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36335
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:51.728984 25578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:51.729333 25578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:51.729518 25578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:51.737926 25578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:51.738062 25578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:51.738268 25578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:51.749917 25578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:51.751482 25578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:51.752955 25578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:51.768566 25586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:51.768590 25584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:51.769024 25583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:51.769414 25578 server_base.cc:1034] running on GCE node
I20260620 14:53:51.770260 25578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:51.771448 25578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:51.772751 25578 hybrid_clock.cc:648] HybridClock initialized: now 1781967231772668 us; error 73 us; skew 500 ppm
I20260620 14:53:51.773131 25578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:51.777113 25578 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:51.779204 25578 webserver.cc:460] Webserver started at http://127.21.159.195:35625/ using document root <none> and password file <none>
I20260620 14:53:51.779959 25578 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:51.780076 25578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:51.780357 25578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:51.783816 25578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "6da09eab5f254d09b7df153caa3eec10"
format_stamp: "Formatted at 2026-06-20 14:53:51 on dist-test-slave-3mqx"
server_key: "fbfd933a8461467f2b4987decea11173"
server_key_iv: "dab38f0f6a6d8bc75410efe10e612d08"
server_key_version: "encryptionkey@0"
I20260620 14:53:51.784801 25578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "6da09eab5f254d09b7df153caa3eec10"
format_stamp: "Formatted at 2026-06-20 14:53:51 on dist-test-slave-3mqx"
server_key: "fbfd933a8461467f2b4987decea11173"
server_key_iv: "dab38f0f6a6d8bc75410efe10e612d08"
server_key_version: "encryptionkey@0"
I20260620 14:53:51.790792 25578 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:53:51.794900 25593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:51.795271 25438 heartbeater.cc:503] Master 127.21.159.254:36335 was elected leader, sending a full tablet report...
I20260620 14:53:51.796790 25578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:53:51.797005 25578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "6da09eab5f254d09b7df153caa3eec10"
format_stamp: "Formatted at 2026-06-20 14:53:51 on dist-test-slave-3mqx"
server_key: "fbfd933a8461467f2b4987decea11173"
server_key_iv: "dab38f0f6a6d8bc75410efe10e612d08"
server_key_version: "encryptionkey@0"
I20260620 14:53:51.797242 25578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:51.810606 25578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:51.815400 25578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:51.815671 25578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:51.816759 25578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:51.818132 25578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:51.818207 25578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:51.818288 25578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:51.818323 25578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:51.853178 25578 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:44307
I20260620 14:53:51.853241 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:44307 every 8 connection(s)
I20260620 14:53:51.854542 25578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:51.857564 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25578
I20260620 14:53:51.857772 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:51.858300 22143 external_mini_cluster.cc:1428] Setting key d1d7b910ae4b6c550163adf4e48b3b59
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:51.877245 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.857433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:38321 (local address 127.21.159.254:36335)
0620 14:53:51.858021 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:51.858028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:51.859181 (+ 1153us) server_negotiation.cc:407] Connection header received
0620 14:53:51.861623 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:51.861627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:51.861695 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:51.861784 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:51.863500 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.864280 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.866056 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.866484 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.866585 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:51.870643 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:51.870649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:51.870652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:51.870701 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:51.873770 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.874815 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.874819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.874821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.874939 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.875537 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.875540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.875542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.875811 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:51.875927 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:51.876808 (+ 881us) server_negotiation.cc:299] Negotiation successful
0620 14:53:51.876980 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":298,"threads_started":1}
I20260620 14:53:51.878485 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.858079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36335 (local address 127.21.159.195:38321)
0620 14:53:51.858976 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:51.859022 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:51.861305 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:51.862012 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:51.862020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:51.862571 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:51.863335 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:51.863356 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.864479 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.864483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:51.865893 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:51.865911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.866713 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.866718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:51.866992 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:51.867854 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:51.867878 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:51.870331 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:51.873947 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:51.873958 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:51.873980 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:51.874684 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:51.875167 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:51.875172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:51.875175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:51.875424 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:51.875958 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:51.875975 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:51.876428 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 14:53:51.876725 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 14:53:51.877203 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":339,"threads_started":1}
I20260620 14:53:51.880147 25706 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36335
I20260620 14:53:51.880507 25706 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:51.881300 25706 heartbeater.cc:511] Master 127.21.159.254:36335 requested a full tablet report, sending...
I20260620 14:53:51.883116 25249 ts_manager.cc:194] Registered new tserver with Master: 6da09eab5f254d09b7df153caa3eec10 (127.21.159.195:44307)
I20260620 14:53:51.884131 25249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:38321
I20260620 14:53:51.887387 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:51.941139 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.924750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.21.159.254:36335)
0620 14:53:51.925045 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:51.925053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:51.925367 (+ 314us) server_negotiation.cc:407] Connection header received
0620 14:53:51.925518 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:51.925522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:51.925614 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:51.925737 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:51.927708 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.928584 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.930092 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.930588 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.930733 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:51.935154 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:51.935161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:51.935164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:51.935206 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:51.938022 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.938832 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.938837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.938840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.938984 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.939498 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.939502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.939504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.939931 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:51.940143 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:51.940524 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 14:53:51.940782 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260620 14:53:51.941936 25249 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43624
I20260620 14:53:51.944239 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25310
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:51.962790 25718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.945422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.21.159.193:46495)
0620 14:53:51.946219 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:51.946227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:51.946253 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:53:51.946315 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:51.946322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:51.946564 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:51.946798 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:51.948375 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.949446 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.950748 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.951448 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.951552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:51.955987 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:51.956010 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:51.956034 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:51.956088 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:51.959617 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.960423 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.960432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.960434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.960575 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.961076 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.961084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.961092 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.961489 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:51.961829 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:51.962149 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:53:51.962413 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":413,"threads_started":1}
W20260620 14:53:51.964174 25433 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55384
W20260620 14:53:51.965353 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25310: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:51.965595 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25310
I20260620 14:53:51.981489 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25444
Jun 20 14:53:51 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:53:51.998965 25719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.982440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.21.159.194:43007)
0620 14:53:51.983045 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:51.983114 (+ 69us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:51.983150 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:53:51.983222 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:51.983229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:51.983459 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:51.983597 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:51.985228 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.986390 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.987774 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:51.988316 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:51.988413 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:51.992826 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:51.992840 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:51.992856 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:51.992896 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:51.996034 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.996698 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.996705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.996708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.996829 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:51.997332 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:51.997336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:51.997339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:51.997763 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:51.998018 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:51.998386 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:53:51.998611 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":387,"threads_started":1}
W20260620 14:53:52.000787 25567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59858
W20260620 14:53:52.002285 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25444: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:52.002467 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25444
I20260620 14:53:52.020193 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25578
Jun 20 14:53:52 dist-test-slave-3mqx krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967231, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:53:52.039501 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:52.021406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.21.159.195:44307)
0620 14:53:52.022165 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:52.022172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:52.022204 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:53:52.022269 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:52.022276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:52.022514 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:52.022656 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:52.024906 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:52.026010 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:52.027613 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:52.028266 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:52.028388 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:52.033307 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:52.033322 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:52.033338 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:52.033380 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:52.036485 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:52.037265 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:52.037274 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:52.037277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:52.037407 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:52.037847 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:52.037850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:52.037852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:52.038192 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:52.038439 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:52.038830 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:53:52.039037 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":455,"threads_started":1}
W20260620 14:53:52.040840 25701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58190
W20260620 14:53:52.042110 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:52.042354 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25578
I20260620 14:53:52.059948 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25217
I20260620 14:53:52.061333 25279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:52.167102 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25217
2026-06-20T14:53:52Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3917 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:52 dist-test-slave-3mqx krb5kdc[25724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:52 dist-test-slave-3mqx krb5kdc[25724](info): set up 2 sockets
Jun 20 14:53:52 dist-test-slave-3mqx krb5kdc[25724](info): commencing operation
krb5kdc: starting...
W20260620 14:53:53.272083 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:53 dist-test-slave-3mqx krb5kdc[25724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:53.479475 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:45057
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:53.817477 25737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:53.817803 25737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:53.817961 25737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:53.825603 25737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:53.825738 25737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:53.825819 25737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:53.825908 25737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:53.826000 25737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:53.836406 25737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:45057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:53.837841 25737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:53.839326 25737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:53.852036 25742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:53.852051 25743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:53.852250 25745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:53.852813 25737 server_base.cc:1034] running on GCE node
I20260620 14:53:53.853612 25737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:53.855130 25737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:53.856385 25737 hybrid_clock.cc:648] HybridClock initialized: now 1781967233856337 us; error 42 us; skew 500 ppm
I20260620 14:53:53.856771 25737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:53 dist-test-slave-3mqx krb5kdc[25724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967233, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:53.860289 25737 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:53.861901 25737 webserver.cc:460] Webserver started at http://127.21.159.254:38251/ using document root <none> and password file <none>
I20260620 14:53:53.862601 25737 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:53.862704 25737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:53.863008 25737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:53.866071 25737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "25af8ff2efe142c0a9436d2f49e997fd"
format_stamp: "Formatted at 2026-06-20 14:53:53 on dist-test-slave-3mqx"
server_key: "a7ea1b56b86c49654b6e8e745a56fbb6"
server_key_iv: "13dbd8d7c2cc53ea9b6fb39862daff52"
server_key_version: "encryptionkey@0"
I20260620 14:53:53.866806 25737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "25af8ff2efe142c0a9436d2f49e997fd"
format_stamp: "Formatted at 2026-06-20 14:53:53 on dist-test-slave-3mqx"
server_key: "a7ea1b56b86c49654b6e8e745a56fbb6"
server_key_iv: "13dbd8d7c2cc53ea9b6fb39862daff52"
server_key_version: "encryptionkey@0"
I20260620 14:53:53.872309 25737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 14:53:53.876022 25752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:53.877766 25737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:53:53.877976 25737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "25af8ff2efe142c0a9436d2f49e997fd"
format_stamp: "Formatted at 2026-06-20 14:53:53 on dist-test-slave-3mqx"
server_key: "a7ea1b56b86c49654b6e8e745a56fbb6"
server_key_iv: "13dbd8d7c2cc53ea9b6fb39862daff52"
server_key_version: "encryptionkey@0"
I20260620 14:53:53.878218 25737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:53.907395 25737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:53.912674 25737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:53.913018 25737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:53.934326 25737 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:45057
I20260620 14:53:53.934370 25803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:45057 every 8 connection(s)
I20260620 14:53:53.935942 25737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:53.940173 25804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:53.940629 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25737
I20260620 14:53:53.940932 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:53.941632 22143 external_mini_cluster.cc:1428] Setting key 8dc0317c9246634f6144a45e707cd19c
I20260620 14:53:53.955682 25804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd: Bootstrap starting.
Jun 20 14:53:53 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:53.958827 25804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:53.960080 25804 log.cc:824] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:53.963371 25804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd: No bootstrap required, opened a new log
I20260620 14:53:53.967229 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:53.944059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.21.159.254:45057)
0620 14:53:53.944890 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:53.944907 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:53.944972 (+ 65us) server_negotiation.cc:407] Connection header received
0620 14:53:53.947313 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:53.947351 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:53.947980 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:53.948526 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:53.950253 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:53.952412 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:53.953851 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:53.954952 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:53.955184 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:53.959228 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:53.959237 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:53.959256 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:53.959301 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:53.962666 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:53.963553 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:53.963560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:53.963567 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:53.963741 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:53.964193 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:53.964202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:53.964204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:53.964760 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:53.965125 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:53.965606 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 14:53:53.966060 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":323,"threads_started":1}
I20260620 14:53:53.971690 25804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25af8ff2efe142c0a9436d2f49e997fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45057 } }
I20260620 14:53:53.972154 25804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:53.972298 25804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25af8ff2efe142c0a9436d2f49e997fd, State: Initialized, Role: FOLLOWER
I20260620 14:53:53.972925 25804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25af8ff2efe142c0a9436d2f49e997fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45057 } }
I20260620 14:53:53.973140 25804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:53.973241 25804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:53.973364 25804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:53.975559 25804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25af8ff2efe142c0a9436d2f49e997fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45057 } }
I20260620 14:53:53.976015 25804 leader_election.cc:302] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25af8ff2efe142c0a9436d2f49e997fd; no voters:
I20260620 14:53:53.976552 25804 leader_election.cc:288] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:53.976719 25809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:53.977387 25809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [term 1 LEADER]: Becoming Leader. State: Replica: 25af8ff2efe142c0a9436d2f49e997fd, State: Running, Role: LEADER
I20260620 14:53:53.977984 25809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25af8ff2efe142c0a9436d2f49e997fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45057 } }
I20260620 14:53:53.978811 25804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:53.980829 25811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25af8ff2efe142c0a9436d2f49e997fd. Latest consensus state: current_term: 1 leader_uuid: "25af8ff2efe142c0a9436d2f49e997fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25af8ff2efe142c0a9436d2f49e997fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45057 } } }
I20260620 14:53:53.981194 25811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:53.981909 25810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25af8ff2efe142c0a9436d2f49e997fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25af8ff2efe142c0a9436d2f49e997fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45057 } } }
I20260620 14:53:53.982259 25810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:53.982462 25815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:53.989866 25815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:53.999956 25815 catalog_manager.cc:1347] Generated new cluster ID: f882b5dbe0de4d048f458e7892298f09
I20260620 14:53:54.000077 25815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:54.011319 25815 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:54.013353 25815 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:54.028752 25815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25af8ff2efe142c0a9436d2f49e997fd: Generated new TSK 0
I20260620 14:53:54.029698 25815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:54.081521 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:45057
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:54.431295 25830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:54.431599 25830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:54.431780 25830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:54.439360 25830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:54.439487 25830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:54.439672 25830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:54.450066 25830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:54.451567 25830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:54.452967 25830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:54.466413 25835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:54.466504 25836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:54.466583 25838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:54.467835 25830 server_base.cc:1034] running on GCE node
I20260620 14:53:54.468528 25830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:54.469578 25830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:54.470805 25830 hybrid_clock.cc:648] HybridClock initialized: now 1781967234470705 us; error 91 us; skew 500 ppm
I20260620 14:53:54.471251 25830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:54 dist-test-slave-3mqx krb5kdc[25724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967234, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:54.474696 25830 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:54.476457 25830 webserver.cc:460] Webserver started at http://127.21.159.193:45491/ using document root <none> and password file <none>
I20260620 14:53:54.477219 25830 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:54.477367 25830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:54.477689 25830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:54.481037 25830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "c35e5f14f9b247b583fa406efef7beab"
format_stamp: "Formatted at 2026-06-20 14:53:54 on dist-test-slave-3mqx"
server_key: "62874b911158b49f64c6b39414432cb2"
server_key_iv: "2852cad04a14a6589cea3a9773c33d25"
server_key_version: "encryptionkey@0"
I20260620 14:53:54.481913 25830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "c35e5f14f9b247b583fa406efef7beab"
format_stamp: "Formatted at 2026-06-20 14:53:54 on dist-test-slave-3mqx"
server_key: "62874b911158b49f64c6b39414432cb2"
server_key_iv: "2852cad04a14a6589cea3a9773c33d25"
server_key_version: "encryptionkey@0"
I20260620 14:53:54.487675 25830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:53:54.491446 25845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:54.493057 25830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:53:54.493242 25830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "c35e5f14f9b247b583fa406efef7beab"
format_stamp: "Formatted at 2026-06-20 14:53:54 on dist-test-slave-3mqx"
server_key: "62874b911158b49f64c6b39414432cb2"
server_key_iv: "2852cad04a14a6589cea3a9773c33d25"
server_key_version: "encryptionkey@0"
I20260620 14:53:54.493433 25830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:54.522571 25830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:54.527293 25830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:54.527637 25830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:54.528756 25830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:54.530166 25830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:54.530243 25830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:54.530370 25830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:54.530435 25830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:54.566056 25830 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:41519
I20260620 14:53:54.566148 25957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:41519 every 8 connection(s)
I20260620 14:53:54.567422 25830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:54.568749 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25830
I20260620 14:53:54.568987 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:54.569808 22143 external_mini_cluster.cc:1428] Setting key 48ad61bb3b729eb54eec99be3e690698
Jun 20 14:53:54 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967234, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:53:54.593672 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:54.570582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:42729 (local address 127.21.159.254:45057)
0620 14:53:54.571140 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:54.571145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:54.572416 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 14:53:54.574363 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:54.574369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:54.574471 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:54.574609 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:54.576283 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:54.577415 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:54.579229 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:54.580007 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:54.580157 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:54.584863 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:54.584874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:54.584882 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:54.584934 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:54.589253 (+ 4319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:54.590305 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:54.590314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:54.590317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:54.590517 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:54.591236 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:54.591241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:54.591243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:54.591647 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:54.591774 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:54.592809 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 14:53:54.593173 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":250,"threads_started":1}
I20260620 14:53:54.594936 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:54.571331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:45057 (local address 127.21.159.193:42729)
0620 14:53:54.572258 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:54.572305 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:54.574068 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:54.574862 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:54.574870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:54.575504 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:54.576097 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:54.576114 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:54.577605 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:54.577609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:54.578983 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:54.579031 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:54.580319 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:54.580327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:54.580559 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:54.581457 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:54.581487 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:54.584511 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:54.589506 (+ 4995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:54.589515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:54.589533 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:54.590099 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:54.590674 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:54.590687 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:54.590690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:54.591021 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:54.591810 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:54.591818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:54.592328 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 14:53:54.592641 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 14:53:54.593193 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":343,"threads_started":1}
I20260620 14:53:54.596979 25958 heartbeater.cc:347] Connected to a master server at 127.21.159.254:45057
I20260620 14:53:54.597465 25958 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:54.598382 25958 heartbeater.cc:511] Master 127.21.159.254:45057 requested a full tablet report, sending...
I20260620 14:53:54.600852 25769 ts_manager.cc:194] Registered new tserver with Master: c35e5f14f9b247b583fa406efef7beab (127.21.159.193:41519)
I20260620 14:53:54.602355 25769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:42729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:54.622422 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:45057
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:54.961990 25964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:54.962345 25964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:54.962524 25964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:54.970191 25964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:54.970386 25964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:54.970590 25964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:54.980808 25964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:54.982288 25964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:54.983824 25964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:54.997287 25970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:54.997341 25969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:54.997293 25972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:54.998872 25964 server_base.cc:1034] running on GCE node
I20260620 14:53:54.999570 25964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:55.000684 25964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:55.001942 25964 hybrid_clock.cc:648] HybridClock initialized: now 1781967235001886 us; error 43 us; skew 500 ppm
I20260620 14:53:55.002326 25964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:54 dist-test-slave-3mqx krb5kdc[25724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967234, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:55.005895 25964 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:55.007643 25964 webserver.cc:460] Webserver started at http://127.21.159.194:36245/ using document root <none> and password file <none>
I20260620 14:53:55.008365 25964 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:55.008469 25964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:55.008747 25964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:55.011799 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "d0d5c9bfb1e3484381c2ecc1ff1fdf68"
format_stamp: "Formatted at 2026-06-20 14:53:55 on dist-test-slave-3mqx"
server_key: "516a5c031eb8ad13a42b52ab6838d082"
server_key_iv: "dad41a047ec9585988cfeda206c7f1f5"
server_key_version: "encryptionkey@0"
I20260620 14:53:55.012656 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "d0d5c9bfb1e3484381c2ecc1ff1fdf68"
format_stamp: "Formatted at 2026-06-20 14:53:55 on dist-test-slave-3mqx"
server_key: "516a5c031eb8ad13a42b52ab6838d082"
server_key_iv: "dad41a047ec9585988cfeda206c7f1f5"
server_key_version: "encryptionkey@0"
I20260620 14:53:55.018209 25964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:53:55.021953 25979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:55.023667 25964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:55.023862 25964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "d0d5c9bfb1e3484381c2ecc1ff1fdf68"
format_stamp: "Formatted at 2026-06-20 14:53:55 on dist-test-slave-3mqx"
server_key: "516a5c031eb8ad13a42b52ab6838d082"
server_key_iv: "dad41a047ec9585988cfeda206c7f1f5"
server_key_version: "encryptionkey@0"
I20260620 14:53:55.024083 25964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:55.041121 25964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:55.045986 25964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:55.046350 25964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:55.047494 25964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:55.049386 25964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:55.049492 25964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:55.049660 25964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:55.049743 25964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:55.086055 25964 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:44439
I20260620 14:53:55.086104 26091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:44439 every 8 connection(s)
I20260620 14:53:55.087522 25964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:55.094311 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 25964
I20260620 14:53:55.094564 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:55.095008 22143 external_mini_cluster.cc:1428] Setting key 7b407629349287398e0178814212faa8
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967234, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:55.110836 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.090264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:32847 (local address 127.21.159.254:45057)
0620 14:53:55.090438 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:55.090442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:55.091996 (+ 1554us) server_negotiation.cc:407] Connection header received
0620 14:53:55.094101 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:55.094111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:55.094216 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:55.094372 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:55.096430 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.097351 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.099478 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.100239 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.100399 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.104569 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:55.104576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:55.104581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:55.104631 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:55.107577 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.108402 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.108410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.108418 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.108549 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.109028 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.109032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.109035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.109313 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:55.109475 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:55.110241 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 14:53:55.110479 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:53:55.111948 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.090846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:45057 (local address 127.21.159.194:32847)
0620 14:53:55.091791 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:55.091843 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:55.093741 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:55.094753 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:55.094763 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:55.095472 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:55.096208 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:55.096254 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.097537 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.097542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:55.099316 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:55.099332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.100520 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.100526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:55.100795 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.101581 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:55.101606 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:55.104202 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:55.107757 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:55.107768 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:55.107790 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:55.108247 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:55.108669 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:55.108675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:55.108678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:55.108890 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:55.109448 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:55.109457 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:55.109860 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 14:53:55.110148 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 14:53:55.110733 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":449,"threads_started":1}
I20260620 14:53:55.113739 26092 heartbeater.cc:347] Connected to a master server at 127.21.159.254:45057
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:53:55.114084 26092 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:55.114884 26092 heartbeater.cc:511] Master 127.21.159.254:45057 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:53:55.116678 25769 ts_manager.cc:194] Registered new tserver with Master: d0d5c9bfb1e3484381c2ecc1ff1fdf68 (127.21.159.194:44439)
I20260620 14:53:55.117547 25769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:32847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:55.140494 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:45057
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:55.485914 26097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:55.486266 26097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:55.486442 26097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:55.493965 26097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:55.494097 26097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:55.494305 26097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:55.504756 26097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:45057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:55.506258 26097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:55.507614 26097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:55.521070 26103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:55.521325 26102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:55.521389 26105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:55.522177 26097 server_base.cc:1034] running on GCE node
I20260620 14:53:55.522749 26097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:55.523794 26097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:55.525008 26097 hybrid_clock.cc:648] HybridClock initialized: now 1781967235524954 us; error 33 us; skew 500 ppm
I20260620 14:53:55.525297 26097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:55.528579 26097 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:55.530136 26097 webserver.cc:460] Webserver started at http://127.21.159.195:34401/ using document root <none> and password file <none>
I20260620 14:53:55.530856 26097 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:55.530987 26097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:55.531394 26097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:55.534386 26097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "de02c5d5b53b464684fb529cef4c31ba"
format_stamp: "Formatted at 2026-06-20 14:53:55 on dist-test-slave-3mqx"
server_key: "b9313091c7f29a09aa3a38bd7bacd0c4"
server_key_iv: "d96cff7b193a5ec5c8a52568113149fa"
server_key_version: "encryptionkey@0"
I20260620 14:53:55.535269 26097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "de02c5d5b53b464684fb529cef4c31ba"
format_stamp: "Formatted at 2026-06-20 14:53:55 on dist-test-slave-3mqx"
server_key: "b9313091c7f29a09aa3a38bd7bacd0c4"
server_key_iv: "d96cff7b193a5ec5c8a52568113149fa"
server_key_version: "encryptionkey@0"
I20260620 14:53:55.540781 26097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 14:53:55.545300 26112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:55.546963 26097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:53:55.547243 26097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "de02c5d5b53b464684fb529cef4c31ba"
format_stamp: "Formatted at 2026-06-20 14:53:55 on dist-test-slave-3mqx"
server_key: "b9313091c7f29a09aa3a38bd7bacd0c4"
server_key_iv: "d96cff7b193a5ec5c8a52568113149fa"
server_key_version: "encryptionkey@0"
I20260620 14:53:55.547519 26097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:55.565035 26097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:55.570405 26097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:55.570739 26097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:55.571825 26097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:55.573307 26097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:55.573432 26097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:55.573611 26097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:55.573702 26097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:55.605841 25958 heartbeater.cc:503] Master 127.21.159.254:45057 was elected leader, sending a full tablet report...
I20260620 14:53:55.611765 26097 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:46279
I20260620 14:53:55.611800 26224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:46279 every 8 connection(s)
I20260620 14:53:55.613229 26097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:55.620837 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 26097
I20260620 14:53:55.621037 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:55.621429 22143 external_mini_cluster.cc:1428] Setting key 931b1abbedd8b023801012975186faee
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967235, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:55.634299 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.615886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:58035 (local address 127.21.159.254:45057)
0620 14:53:55.616297 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:55.616302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:55.617427 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 14:53:55.619251 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:55.619255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:55.619323 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:55.619425 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:55.621141 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.622041 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.624378 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.624903 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.625004 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.628674 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:55.628679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:55.628682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:55.628717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:55.631477 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.632186 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.632190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.632192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.632307 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.632777 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.632780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.632782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.633031 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:55.633144 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:55.633870 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 14:53:55.634030 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":220,"threads_started":1}
I20260620 14:53:55.635483 26228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.616549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:45057 (local address 127.21.159.195:58035)
0620 14:53:55.617257 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:55.617294 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:55.618914 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:55.619712 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:55.619722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:55.620283 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:55.620889 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:55.620959 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.622247 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.622251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:55.624214 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:55.624230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.625103 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.625108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:55.625315 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.625971 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:55.625992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:55.628474 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:55.631615 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:55.631623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:55.631637 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:55.632073 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:55.632429 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:55.632432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:55.632435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:55.632670 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:55.633145 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:55.633150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:55.633531 (+ 381us) client_negotiation.cc:769] Sending connection context
0620 14:53:55.633777 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:53:55.634173 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":265,"threads_started":1}
I20260620 14:53:55.636973 26225 heartbeater.cc:347] Connected to a master server at 127.21.159.254:45057
I20260620 14:53:55.637310 26225 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:55.638064 26225 heartbeater.cc:511] Master 127.21.159.254:45057 requested a full tablet report, sending...
I20260620 14:53:55.639603 25769 ts_manager.cc:194] Registered new tserver with Master: de02c5d5b53b464684fb529cef4c31ba (127.21.159.195:46279)
I20260620 14:53:55.640396 25769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:58035
I20260620 14:53:55.649780 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:53:55.700336 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.684293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.21.159.193:41519)
0620 14:53:55.684998 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:55.685006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:55.685046 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:53:55.685107 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:55.685114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:55.685383 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:55.685556 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:55.687150 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.688189 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.689434 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.689991 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.690109 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.694279 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:55.694291 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:55.694306 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:55.694347 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:55.697458 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.698214 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.698224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.698227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.698398 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.698828 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.698832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.698834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.699272 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:55.699502 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:55.699844 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:53:55.700050 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":368,"threads_started":1}
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:55.718964 26227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.704990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.21.159.254:45057)
0620 14:53:55.705206 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:55.705213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:55.705342 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:53:55.705447 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:55.705450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:55.705514 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:55.705602 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:55.706894 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.707735 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.709026 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.709441 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.709580 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.713531 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:55.713536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:55.713539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:55.713575 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:55.716283 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.717046 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.717050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.717052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.717167 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.717622 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.717625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.717627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.717940 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:55.718120 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:55.718452 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 14:53:55.718651 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 14:53:55.719658 25769 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49706
I20260620 14:53:55.720809 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25830
I20260620 14:53:55.734299 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.721699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.21.159.193:41519)
0620 14:53:55.721866 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:55.721870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:55.722196 (+ 326us) server_negotiation.cc:407] Connection header received
0620 14:53:55.722327 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:55.722331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:55.722420 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:55.722500 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:55.724041 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.724734 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.725875 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.726284 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.726379 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.728930 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:55.728935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:55.728938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:55.728974 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:55.731755 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.732504 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.732508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.732511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.732652 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.733027 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.733030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.733032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.733302 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:55.733476 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:55.733882 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 14:53:55.734052 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:53:55.734982 25953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:55.818038 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25830
I20260620 14:53:55.835981 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25964
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:53:55.852792 26238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.836987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.21.159.194:44439)
0620 14:53:55.837527 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:55.837534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:55.837562 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:53:55.837635 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:55.837641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:55.837861 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:55.838086 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:55.839366 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.840435 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.841702 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.842276 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.842416 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.846681 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:55.846693 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:55.846709 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:55.846743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:55.849595 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.850358 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.850366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.850368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.850496 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.850891 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.850897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.850901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.851464 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:55.851779 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:55.852142 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:53:55.852410 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":279,"threads_started":1}
I20260620 14:53:55.854177 26087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:55.938565 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25964
I20260620 14:53:55.955679 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 26097
Jun 20 14:53:55 dist-test-slave-3mqx krb5kdc[25724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:53:55.973414 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.956763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.21.159.195:46279)
0620 14:53:55.957448 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:55.957456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:55.957487 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:53:55.957550 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:55.957556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:55.957771 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:55.957959 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:55.959352 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.960422 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.961766 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:55.962363 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:55.962516 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:55.967099 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:55.967118 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:55.967134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:55.967173 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:55.970338 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.971213 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.971221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.971224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.971355 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:55.971756 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:55.971766 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:55.971768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:55.972136 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:55.972461 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:55.972790 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 14:53:55.973066 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":296,"threads_started":1}
I20260620 14:53:55.974769 26220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:56.060482 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 26097
I20260620 14:53:56.077507 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 25737
I20260620 14:53:56.078734 25799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:56.181957 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 25737
2026-06-20T14:53:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4017 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:56 dist-test-slave-3mqx krb5kdc[26245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:56 dist-test-slave-3mqx krb5kdc[26245](info): set up 2 sockets
Jun 20 14:53:56 dist-test-slave-3mqx krb5kdc[26245](info): commencing operation
krb5kdc: starting...
W20260620 14:53:57.287703 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:53:57 dist-test-slave-3mqx krb5kdc[26245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:53:57.498203 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:43821
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:57.834746 26258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:57.835165 26258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:57.835356 26258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:57.843010 26258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:57.843225 26258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:57.843313 26258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:57.843417 26258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:57.843489 26258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:57.853785 26258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:57.855335 26258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:57.856747 26258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:57.869099 26263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:57.869165 26264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:57.869143 26266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:57.870486 26258 server_base.cc:1034] running on GCE node
I20260620 14:53:57.871475 26258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:57.873098 26258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:57.874348 26258 hybrid_clock.cc:648] HybridClock initialized: now 1781967237874266 us; error 65 us; skew 500 ppm
I20260620 14:53:57.874881 26258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:57 dist-test-slave-3mqx krb5kdc[26245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:57.883163 26258 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:53:57.884629 26258 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:57.884745 26258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:57.885074 26258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:57.888736 26258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "92cd833442d34970813dd4c02784b2ff"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-3mqx"
server_key: "bcc705db28ff2bfd2ed808d4287e247b"
server_key_iv: "5e5fbaf9f16e0c37d9656eff6bbb98d6"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.889678 26258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "92cd833442d34970813dd4c02784b2ff"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-3mqx"
server_key: "bcc705db28ff2bfd2ed808d4287e247b"
server_key_iv: "5e5fbaf9f16e0c37d9656eff6bbb98d6"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.895925 26258 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:53:57.900286 26272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.902068 26258 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:53:57.902264 26258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "92cd833442d34970813dd4c02784b2ff"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-3mqx"
server_key: "bcc705db28ff2bfd2ed808d4287e247b"
server_key_iv: "5e5fbaf9f16e0c37d9656eff6bbb98d6"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.902493 26258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:57.938184 26258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:57.943529 26258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:57.943871 26258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:57.964505 26258 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:43821
I20260620 14:53:57.964474 26323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:43821 every 8 connection(s)
I20260620 14:53:57.965588 26258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:57.969367 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 26258
I20260620 14:53:57.969635 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:53:57.970054 26324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:57.970467 22143 external_mini_cluster.cc:1428] Setting key 96ed2ff102d501d704f222fe02540e51
I20260620 14:53:57.987039 26324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff: Bootstrap starting.
Jun 20 14:53:57 dist-test-slave-3mqx krb5kdc[26245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:57.990873 26324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:57.992156 26324 log.cc:824] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:57.995443 26324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff: No bootstrap required, opened a new log
I20260620 14:53:57.998152 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.972718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.21.159.254:43821)
0620 14:53:57.973636 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.973650 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.973696 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:53:57.975355 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.975382 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.976097 (+ 715us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.976585 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:57.978993 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.980666 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.982811 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.983862 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.984033 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.989517 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:57.989526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:57.989544 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:57.989608 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:57.993282 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.994217 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.994226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.994233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.994413 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.994866 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.994873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.994876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.995665 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:57.995990 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.996498 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.996989 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":434,"threads_started":1}
I20260620 14:53:58.005002 26324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cd833442d34970813dd4c02784b2ff" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43821 } }
I20260620 14:53:58.005594 26324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:58.005771 26324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cd833442d34970813dd4c02784b2ff, State: Initialized, Role: FOLLOWER
I20260620 14:53:58.006556 26324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cd833442d34970813dd4c02784b2ff" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43821 } }
I20260620 14:53:58.006870 26324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:58.007038 26324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:58.007329 26324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:58.009827 26324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cd833442d34970813dd4c02784b2ff" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43821 } }
I20260620 14:53:58.010478 26324 leader_election.cc:302] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92cd833442d34970813dd4c02784b2ff; no voters:
I20260620 14:53:58.011242 26324 leader_election.cc:288] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:58.011391 26329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:58.012193 26329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [term 1 LEADER]: Becoming Leader. State: Replica: 92cd833442d34970813dd4c02784b2ff, State: Running, Role: LEADER
I20260620 14:53:58.012723 26329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cd833442d34970813dd4c02784b2ff" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43821 } }
I20260620 14:53:58.013833 26324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:58.016618 26330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92cd833442d34970813dd4c02784b2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cd833442d34970813dd4c02784b2ff" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43821 } } }
I20260620 14:53:58.017326 26330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:58.019805 26331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92cd833442d34970813dd4c02784b2ff. Latest consensus state: current_term: 1 leader_uuid: "92cd833442d34970813dd4c02784b2ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92cd833442d34970813dd4c02784b2ff" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43821 } } }
I20260620 14:53:58.020304 26331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:58.020586 26338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:58.026388 26338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:58.036563 26338 catalog_manager.cc:1347] Generated new cluster ID: 800bea69e8fe4169a4b049e83c4efebc
I20260620 14:53:58.036700 26338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:58.063372 26338 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:58.064981 26338 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:58.075150 26338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff: Generated new TSK 0
I20260620 14:53:58.076206 26338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:53:58.138518 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43821
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:58.486722 26350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:58.487147 26350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:58.487272 26350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:58.495780 26350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:58.495919 26350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:58.496055 26350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:53:58.507987 26350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:58.509449 26350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:58.510844 26350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:58.526389 26358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:58.526495 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:58.526475 26356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:58.532727 26350 server_base.cc:1034] running on GCE node
I20260620 14:53:58.533463 26350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:58.534534 26350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:58.535774 26350 hybrid_clock.cc:648] HybridClock initialized: now 1781967238535726 us; error 66 us; skew 500 ppm
I20260620 14:53:58.536123 26350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:58 dist-test-slave-3mqx krb5kdc[26245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:58.539530 26350 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:53:58.540809 26350 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:58.540902 26350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:58.541175 26350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:58.544303 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9"
format_stamp: "Formatted at 2026-06-20 14:53:58 on dist-test-slave-3mqx"
server_key: "2e6df32eaffc2098c09640b9d015f2bb"
server_key_iv: "35ac2eb9062ba9f9e601b093ee56aa9e"
server_key_version: "encryptionkey@0"
I20260620 14:53:58.545110 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9"
format_stamp: "Formatted at 2026-06-20 14:53:58 on dist-test-slave-3mqx"
server_key: "2e6df32eaffc2098c09640b9d015f2bb"
server_key_iv: "35ac2eb9062ba9f9e601b093ee56aa9e"
server_key_version: "encryptionkey@0"
I20260620 14:53:58.550485 26350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:53:58.554239 26364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:58.555908 26350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:58.556121 26350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9"
format_stamp: "Formatted at 2026-06-20 14:53:58 on dist-test-slave-3mqx"
server_key: "2e6df32eaffc2098c09640b9d015f2bb"
server_key_iv: "35ac2eb9062ba9f9e601b093ee56aa9e"
server_key_version: "encryptionkey@0"
I20260620 14:53:58.556372 26350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:58.583870 26350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:58.588601 26350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:58.588918 26350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:58.590314 26350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:58.592033 26350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:58.592141 26350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:58.592306 26350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:58.592399 26350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:58.628892 26350 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:40215
I20260620 14:53:58.628988 26476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:40215 every 8 connection(s)
I20260620 14:53:58.629859 26350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:58.632998 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 26350
I20260620 14:53:58.633252 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:58.633939 22143 external_mini_cluster.cc:1428] Setting key 0447d90485d60ab2eabc6a93fa3fd891
Jun 20 14:53:58 dist-test-slave-3mqx krb5kdc[26245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967238, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:53:58.653261 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:58.632631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:58381 (local address 127.21.159.254:43821)
0620 14:53:58.633300 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:58.633309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:58.634341 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 14:53:58.636483 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:58.636490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:58.636602 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:58.636741 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:58.638539 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:58.639866 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:58.641620 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:58.642039 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:58.642184 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:58.646772 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:58.646778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:58.646782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:58.646819 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:58.649744 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:58.650628 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:58.650640 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:58.650643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:58.650770 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:58.651331 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:58.651339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:58.651343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:58.651637 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:58.651808 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:58.652616 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 14:53:58.652866 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":366,"threads_started":1}
I20260620 14:53:58.654533 26480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:58.633296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:43821 (local address 127.21.159.193:58381)
0620 14:53:58.634136 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:58.634190 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:58.636202 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:58.637071 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:58.637081 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:58.637678 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:58.638332 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:58.638354 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:58.640112 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:58.640117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:58.641429 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:58.641443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:58.642296 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:58.642303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:58.642632 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:58.643548 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:58.643584 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:58.646495 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:58.649940 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:58.649952 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:58.649973 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:58.650478 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:58.650898 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:58.650904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:58.650908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:58.651213 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:58.651782 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:58.651792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:58.652205 (+ 413us) client_negotiation.cc:769] Sending connection context
0620 14:53:58.652518 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 14:53:58.653131 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":352,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:53:58.656534 26477 heartbeater.cc:347] Connected to a master server at 127.21.159.254:43821
I20260620 14:53:58.656883 26477 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:58.657681 26477 heartbeater.cc:511] Master 127.21.159.254:43821 requested a full tablet report, sending...
I20260620 14:53:58.660269 26289 ts_manager.cc:194] Registered new tserver with Master: cd56d6ae628f402f9e6ca3f2f430a3b9 (127.21.159.193:40215)
I20260620 14:53:58.662397 26289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:58381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:53:58.682328 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43821
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:59.018806 26483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:59.019162 26483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:59.019261 26483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:59.026620 26483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:59.026741 26483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:59.026861 26483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:53:59.037058 26483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:59.038370 26483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:59.039678 26483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:59.053750 26489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:59.053793 26488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:59.053817 26491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:59.054979 26483 server_base.cc:1034] running on GCE node
I20260620 14:53:59.055778 26483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:59.056862 26483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:59.058116 26483 hybrid_clock.cc:648] HybridClock initialized: now 1781967239058053 us; error 56 us; skew 500 ppm
I20260620 14:53:59.058511 26483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:59 dist-test-slave-3mqx krb5kdc[26245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:59.062268 26483 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:53:59.063668 26483 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:59.063814 26483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:59.064155 26483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:59.067298 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "eb8c15695f96454e8b58d4f4da640347"
format_stamp: "Formatted at 2026-06-20 14:53:59 on dist-test-slave-3mqx"
server_key: "163a650ae0ddeb85216e6893b8611e44"
server_key_iv: "9b95f08cad7e582650cd37e1c70c54a3"
server_key_version: "encryptionkey@0"
I20260620 14:53:59.068212 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "eb8c15695f96454e8b58d4f4da640347"
format_stamp: "Formatted at 2026-06-20 14:53:59 on dist-test-slave-3mqx"
server_key: "163a650ae0ddeb85216e6893b8611e44"
server_key_iv: "9b95f08cad7e582650cd37e1c70c54a3"
server_key_version: "encryptionkey@0"
I20260620 14:53:59.073858 26483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 14:53:59.078372 26497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:59.080241 26483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 14:53:59.080493 26483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "eb8c15695f96454e8b58d4f4da640347"
format_stamp: "Formatted at 2026-06-20 14:53:59 on dist-test-slave-3mqx"
server_key: "163a650ae0ddeb85216e6893b8611e44"
server_key_iv: "9b95f08cad7e582650cd37e1c70c54a3"
server_key_version: "encryptionkey@0"
I20260620 14:53:59.080771 26483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:59.104750 26483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:59.109758 26483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:59.110107 26483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:59.111284 26483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:59.112829 26483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:59.112955 26483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:59.113103 26483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:59.113193 26483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:59.148501 26483 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:36327
I20260620 14:53:59.148588 26609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:36327 every 8 connection(s)
I20260620 14:53:59.149447 26483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:59.154274 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 26483
I20260620 14:53:59.154501 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:59.155110 22143 external_mini_cluster.cc:1428] Setting key 3c104f20caf7c1af0b4442b9924b346e
Jun 20 14:53:59 dist-test-slave-3mqx krb5kdc[26245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:59.174728 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.152220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:33585 (local address 127.21.159.254:43821)
0620 14:53:59.152408 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.152413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.153680 (+ 1267us) server_negotiation.cc:407] Connection header received
0620 14:53:59.156288 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.156294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.156394 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.156512 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:59.158601 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.159402 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.162155 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.162604 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.162717 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.166770 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:59.166779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:59.166788 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:59.166838 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:59.170714 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:59.171804 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:59.171814 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:59.171821 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:59.172002 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:59.172566 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:59.172574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:59.172581 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:59.173021 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:59.173201 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.174016 (+ 815us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.174314 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:53:59.175577 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.152718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:43821 (local address 127.21.159.194:33585)
0620 14:53:59.153485 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:59.153531 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:59.155955 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:59.156771 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:59.156781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:59.157460 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:59.158376 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.158408 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.159572 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.159580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.161958 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.161981 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.162880 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.162890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.163295 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.164126 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:59.164151 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:59.166475 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:59.170934 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:59.170942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:59.170959 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:59.171626 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:59.172201 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:59.172205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:59.172207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:59.172408 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:59.173215 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:59.173221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:59.173601 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 14:53:59.173853 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 14:53:59.174317 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:53:59.177348 26610 heartbeater.cc:347] Connected to a master server at 127.21.159.254:43821
I20260620 14:53:59.177680 26610 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:53:59.178503 26610 heartbeater.cc:511] Master 127.21.159.254:43821 requested a full tablet report, sending...
I20260620 14:53:59.180413 26289 ts_manager.cc:194] Registered new tserver with Master: eb8c15695f96454e8b58d4f4da640347 (127.21.159.194:36327)
I20260620 14:53:59.181339 26289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:33585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:53:59.203945 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43821
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:59.556454 26615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:59.556777 26615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:59.556989 26615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:59.564831 26615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:59.564960 26615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:59.565140 26615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:53:59.575523 26615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:43821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:59.576951 26615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:59.578332 26615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:59.592846 26623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:59.592963 26621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:59.592949 26620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:59.594132 26615 server_base.cc:1034] running on GCE node
I20260620 14:53:59.594903 26615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:59.596071 26615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:59.597311 26615 hybrid_clock.cc:648] HybridClock initialized: now 1781967239597231 us; error 68 us; skew 500 ppm
I20260620 14:53:59.597731 26615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:59 dist-test-slave-3mqx krb5kdc[26245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:59.601591 26615 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:53:59.603125 26615 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:59.603289 26615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:59.603667 26615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:59.607239 26615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "52411b8c21be44e3b21a18b16d8a98e0"
format_stamp: "Formatted at 2026-06-20 14:53:59 on dist-test-slave-3mqx"
server_key: "a3bb91242662685f3ccfbaa06dd5a016"
server_key_iv: "88469904a1f45ab97a7abdcd5ed396f6"
server_key_version: "encryptionkey@0"
I20260620 14:53:59.608304 26615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "52411b8c21be44e3b21a18b16d8a98e0"
format_stamp: "Formatted at 2026-06-20 14:53:59 on dist-test-slave-3mqx"
server_key: "a3bb91242662685f3ccfbaa06dd5a016"
server_key_iv: "88469904a1f45ab97a7abdcd5ed396f6"
server_key_version: "encryptionkey@0"
I20260620 14:53:59.614398 26615 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:53:59.618722 26629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:59.620869 26615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:53:59.621114 26615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "52411b8c21be44e3b21a18b16d8a98e0"
format_stamp: "Formatted at 2026-06-20 14:53:59 on dist-test-slave-3mqx"
server_key: "a3bb91242662685f3ccfbaa06dd5a016"
server_key_iv: "88469904a1f45ab97a7abdcd5ed396f6"
server_key_version: "encryptionkey@0"
I20260620 14:53:59.621393 26615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:59.641279 26615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:59.646363 26615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:59.646687 26615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:59.647902 26615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:59.649449 26615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:59.649580 26615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:59.649711 26615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:59.649808 26615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:59.666525 26477 heartbeater.cc:503] Master 127.21.159.254:43821 was elected leader, sending a full tablet report...
I20260620 14:53:59.685104 26615 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:46211
I20260620 14:53:59.685159 26741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:46211 every 8 connection(s)
I20260620 14:53:59.686029 26615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:59.696092 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 26615
I20260620 14:53:59.696355 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:59.696798 22143 external_mini_cluster.cc:1428] Setting key 8991bb0e0c48427516e5908a47ff8a3c
Jun 20 14:53:59 dist-test-slave-3mqx krb5kdc[26245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967239, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:53:59.708369 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.688766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:39901 (local address 127.21.159.254:43821)
0620 14:53:59.689253 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.689258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.690272 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 14:53:59.692308 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.692313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.692445 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.692580 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:59.694290 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.695165 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.696960 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.697488 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.697634 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.701820 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:59.701825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:59.701829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:59.701864 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:59.704714 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:59.705602 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:59.705607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:59.705609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:59.705733 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:59.706396 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:59.706400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:59.706402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:59.706710 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:59.706827 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.707864 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.708083 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260620 14:53:59.709404 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.689316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:43821 (local address 127.21.159.195:39901)
0620 14:53:59.690073 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:59.690118 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:59.691970 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:59.692854 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:59.692864 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:59.693435 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:59.694075 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.694099 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.695328 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.695339 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.696795 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.696809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.697788 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.697796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.698089 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.698928 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:59.698952 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:59.701600 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:59.704915 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:59.704923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:59.704946 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:59.705474 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:59.705960 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:59.705966 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:59.705970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:59.706282 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:59.706849 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:59.706856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:59.707441 (+ 585us) client_negotiation.cc:769] Sending connection context
0620 14:53:59.707718 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 14:53:59.708209 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":297,"threads_started":1}
I20260620 14:53:59.711161 26742 heartbeater.cc:347] Connected to a master server at 127.21.159.254:43821
I20260620 14:53:59.711544 26742 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:59.712539 26742 heartbeater.cc:511] Master 127.21.159.254:43821 requested a full tablet report, sending...
I20260620 14:53:59.714288 26289 ts_manager.cc:194] Registered new tserver with Master: 52411b8c21be44e3b21a18b16d8a98e0 (127.21.159.195:46211)
I20260620 14:53:59.715233 26289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:39901
I20260620 14:53:59.726104 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:59.745034 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.731403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.21.159.254:43821)
0620 14:53:59.731678 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.731688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.732319 (+ 631us) server_negotiation.cc:407] Connection header received
0620 14:53:59.732431 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.732435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.732543 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.732671 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:59.734188 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.734962 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.736313 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.736761 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.736865 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.739386 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:59.739391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:59.739395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:59.739435 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:59.742230 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:59.743025 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:59.743030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:59.743032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:59.743236 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:59.743609 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:59.743612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:59.743614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:59.743965 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:59.744150 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.744458 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.744655 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:53:59.749563 26289 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:59.756680 26289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:59.795899 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.771892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34998 (local address 127.21.159.195:46211)
0620 14:53:59.786139 (+ 14247us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.786148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.787243 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 14:53:59.787348 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.787358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.787570 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.787765 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:59.788720 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.790073 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.792589 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.794162 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.794379 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.794574 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.795375 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.795552 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14059,"thread_start_us":14253,"threads_started":1}
I20260620 14:53:59.795972 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.770120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:40215 (local address 127.0.0.1:49666)
0620 14:53:59.771029 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:59.771089 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:59.771270 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:59.786990 (+ 15720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:59.787006 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:59.787084 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:59.787523 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.787536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.789712 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.789716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.792626 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.792646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.794482 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.794491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.794877 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.795048 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 14:53:59.795217 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 14:53:59.795446 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":310,"spinlock_wait_cycles":4352,"thread_start_us":513,"threads_started":1}
I20260620 14:53:59.796906 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.771490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:46211 (local address 127.0.0.1:34998)
0620 14:53:59.786918 (+ 15428us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:59.786937 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:59.787111 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:59.788051 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:59.788057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:59.788109 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:59.788477 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.788490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.790226 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.790231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.792417 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.792429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.794498 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.794503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.794875 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.795048 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:53:59.795224 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 14:53:59.796497 (+ 1273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15308,"mutex_wait_us":1167,"spinlock_wait_cycles":3328,"thread_start_us":13928,"threads_started":1}
I20260620 14:53:59.797029 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.770200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.21.159.193:40215)
0620 14:53:59.786175 (+ 15975us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.786188 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.786230 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:53:59.786361 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.786372 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.786697 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.787745 (+ 1048us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:59.788100 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.789536 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.792803 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.794184 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.794379 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.794574 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.796498 (+ 1924us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.796692 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15725,"thread_start_us":16144,"threads_started":1}
I20260620 14:53:59.797343 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.770680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36327 (local address 127.0.0.1:35236)
0620 14:53:59.785053 (+ 14373us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:59.785086 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:59.785292 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:59.787471 (+ 2179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:59.787480 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:59.787511 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:59.787949 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.787962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.789651 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.789659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.792649 (+ 2990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.792658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.796424 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.796431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.796591 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.796640 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 14:53:59.796736 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:53:59.796887 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14141,"mutex_wait_us":213,"thread_start_us":15987,"threads_started":1}
I20260620 14:53:59.797412 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.771393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.21.159.194:36327)
0620 14:53:59.786808 (+ 15415us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.786817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.786848 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:53:59.786946 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.786954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.787243 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.788162 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:59.788419 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.789497 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.794684 (+ 5187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.796180 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.796361 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.796587 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.796907 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.797099 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15248,"thread_start_us":15080,"threads_started":1}
I20260620 14:53:59.801132 26677 tablet_service.cc:1467] Processing CreateTablet for tablet 5d38c2c195a0445e9ca8f13a21271427 (DEFAULT_TABLE table=test-table [id=8986f58a4ad24649840f02ca0623e801]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:59.801132 26412 tablet_service.cc:1467] Processing CreateTablet for tablet 5d38c2c195a0445e9ca8f13a21271427 (DEFAULT_TABLE table=test-table [id=8986f58a4ad24649840f02ca0623e801]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:59.801131 26545 tablet_service.cc:1467] Processing CreateTablet for tablet 5d38c2c195a0445e9ca8f13a21271427 (DEFAULT_TABLE table=test-table [id=8986f58a4ad24649840f02ca0623e801]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:59.802538 26677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d38c2c195a0445e9ca8f13a21271427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:59.802587 26545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d38c2c195a0445e9ca8f13a21271427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:59.802584 26412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d38c2c195a0445e9ca8f13a21271427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:59.816207 26761 tablet_bootstrap.cc:492] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Bootstrap starting.
I20260620 14:53:59.818507 26763 tablet_bootstrap.cc:492] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Bootstrap starting.
I20260620 14:53:59.820959 26761 tablet_bootstrap.cc:654] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:59.821105 26762 tablet_bootstrap.cc:492] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Bootstrap starting.
I20260620 14:53:59.822357 26761 log.cc:824] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:59.825034 26762 tablet_bootstrap.cc:654] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:59.826148 26761 tablet_bootstrap.cc:492] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: No bootstrap required, opened a new log
I20260620 14:53:59.826449 26761 ts_tablet_manager.cc:1397] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260620 14:53:59.826426 26762 log.cc:824] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:59.828478 26763 tablet_bootstrap.cc:654] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:59.829691 26763 log.cc:824] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:59.830147 26762 tablet_bootstrap.cc:492] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: No bootstrap required, opened a new log
I20260620 14:53:59.830499 26762 ts_tablet_manager.cc:1397] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260620 14:53:59.832669 26763 tablet_bootstrap.cc:492] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: No bootstrap required, opened a new log
I20260620 14:53:59.832966 26763 ts_tablet_manager.cc:1397] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20260620 14:53:59.837383 26761 raft_consensus.cc:348] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.837877 26761 raft_consensus.cc:374] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:59.838020 26761 raft_consensus.cc:729] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52411b8c21be44e3b21a18b16d8a98e0, State: Initialized, Role: FOLLOWER
I20260620 14:53:59.838737 26761 consensus_queue.cc:260] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.840037 26742 heartbeater.cc:503] Master 127.21.159.254:43821 was elected leader, sending a full tablet report...
I20260620 14:53:59.840718 26761 ts_tablet_manager.cc:1428] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 14:53:59.842576 26763 raft_consensus.cc:348] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.842576 26762 raft_consensus.cc:348] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.843097 26762 raft_consensus.cc:374] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:59.843094 26763 raft_consensus.cc:374] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:59.843212 26763 raft_consensus.cc:729] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd56d6ae628f402f9e6ca3f2f430a3b9, State: Initialized, Role: FOLLOWER
I20260620 14:53:59.843214 26762 raft_consensus.cc:729] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb8c15695f96454e8b58d4f4da640347, State: Initialized, Role: FOLLOWER
I20260620 14:53:59.843919 26762 consensus_queue.cc:260] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.843910 26763 consensus_queue.cc:260] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.845562 26610 heartbeater.cc:503] Master 127.21.159.254:43821 was elected leader, sending a full tablet report...
I20260620 14:53:59.845983 26763 ts_tablet_manager.cc:1428] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260620 14:53:59.846227 26762 ts_tablet_manager.cc:1428] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Time spent starting tablet: real 0.016s user 0.007s sys 0.006s
W20260620 14:53:59.882401 26478 tablet.cc:2369] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:53:59.901575 26611 tablet.cc:2369] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:53:59.937674 26743 tablet.cc:2369] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:59.979432 26767 raft_consensus.cc:482] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:59.979799 26767 raft_consensus.cc:504] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.981565 26767 leader_election.cc:288] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd56d6ae628f402f9e6ca3f2f430a3b9 (127.21.159.193:40215), eb8c15695f96454e8b58d4f4da640347 (127.21.159.194:36327)
I20260620 14:53:59.987411 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.982146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:50935 (local address 127.21.159.193:40215)
0620 14:53:59.982343 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.982351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.982374 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:53:59.982649 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.982655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.982733 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.982844 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:59.983361 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.984146 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.985746 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.986582 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.986741 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.986824 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.987133 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.987227 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:53:59.987511 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.981919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:40215 (local address 127.21.159.195:50935)
0620 14:53:59.982288 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:59.982310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:59.982474 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:59.982877 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:59.982880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:59.982902 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:59.983212 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.983222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.984288 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.984294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.985575 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.985590 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.986723 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.986728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.986919 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.986954 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:53:59.987084 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:53:59.987220 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"spinlock_wait_cycles":28928}
I20260620 14:53:59.988653 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.982073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:42893 (local address 127.21.159.194:36327)
0620 14:53:59.982282 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:59.982289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:59.982998 (+ 709us) server_negotiation.cc:407] Connection header received
0620 14:53:59.983216 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:59.983220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:59.983278 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:59.983376 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:59.983875 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.984757 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.986589 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.987792 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.987945 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.988010 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:59.988283 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:53:59.988408 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:53:59.988718 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:59.982059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36327 (local address 127.21.159.195:42893)
0620 14:53:59.982866 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:59.982889 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:59.983028 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:59.983390 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:59.983394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:59.983417 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:59.983716 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.983724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.984898 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.984903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.986440 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:59.986452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:59.987954 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:59.987960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:59.988120 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:59.988141 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:53:59.988258 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:53:59.988388 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":397,"threads_started":1}
I20260620 14:53:59.988577 26432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d38c2c195a0445e9ca8f13a21271427" candidate_uuid: "52411b8c21be44e3b21a18b16d8a98e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" is_pre_election: true
I20260620 14:53:59.989167 26432 raft_consensus.cc:2454] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52411b8c21be44e3b21a18b16d8a98e0 in term 0.
I20260620 14:53:59.989768 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d38c2c195a0445e9ca8f13a21271427" candidate_uuid: "52411b8c21be44e3b21a18b16d8a98e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb8c15695f96454e8b58d4f4da640347" is_pre_election: true
I20260620 14:53:59.989892 26630 leader_election.cc:302] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52411b8c21be44e3b21a18b16d8a98e0, cd56d6ae628f402f9e6ca3f2f430a3b9; no voters:
I20260620 14:53:59.990312 26767 raft_consensus.cc:2781] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:59.990332 26565 raft_consensus.cc:2454] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52411b8c21be44e3b21a18b16d8a98e0 in term 0.
I20260620 14:53:59.990430 26767 raft_consensus.cc:482] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:59.990609 26767 raft_consensus.cc:3037] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:59.992847 26767 raft_consensus.cc:504] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:53:59.993465 26767 leader_election.cc:288] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [CANDIDATE]: Term 1 election: Requested vote from peers cd56d6ae628f402f9e6ca3f2f430a3b9 (127.21.159.193:40215), eb8c15695f96454e8b58d4f4da640347 (127.21.159.194:36327)
I20260620 14:53:59.993938 26565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d38c2c195a0445e9ca8f13a21271427" candidate_uuid: "52411b8c21be44e3b21a18b16d8a98e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb8c15695f96454e8b58d4f4da640347"
I20260620 14:53:59.993959 26432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d38c2c195a0445e9ca8f13a21271427" candidate_uuid: "52411b8c21be44e3b21a18b16d8a98e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9"
I20260620 14:53:59.994144 26565 raft_consensus.cc:3037] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:59.994174 26432 raft_consensus.cc:3037] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:59.996404 26565 raft_consensus.cc:2454] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52411b8c21be44e3b21a18b16d8a98e0 in term 1.
I20260620 14:53:59.996502 26432 raft_consensus.cc:2454] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52411b8c21be44e3b21a18b16d8a98e0 in term 1.
I20260620 14:53:59.996855 26631 leader_election.cc:302] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52411b8c21be44e3b21a18b16d8a98e0, eb8c15695f96454e8b58d4f4da640347; no voters:
I20260620 14:53:59.997161 26767 raft_consensus.cc:2781] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:59.997644 26767 raft_consensus.cc:686] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 1 LEADER]: Becoming Leader. State: Replica: 52411b8c21be44e3b21a18b16d8a98e0, State: Running, Role: LEADER
I20260620 14:53:59.998370 26767 consensus_queue.cc:237] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } }
I20260620 14:54:00.002883 26289 catalog_manager.cc:5495] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52411b8c21be44e3b21a18b16d8a98e0 (127.21.159.195). New cstate: current_term: 1 leader_uuid: "52411b8c21be44e3b21a18b16d8a98e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52411b8c21be44e3b21a18b16d8a98e0" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:00.036532 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:00.031043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.21.159.195:46211)
0620 14:54:00.031271 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:00.031276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:00.031296 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:54:00.031546 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:00.031551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:00.031626 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:00.031791 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:00.032169 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.032924 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.034303 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.034702 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.034833 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:00.035393 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:00.035932 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:00.036144 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:00.036192 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:54:00.036270 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:54:00.044960 26565 raft_consensus.cc:1261] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 1 FOLLOWER]: Refusing update from remote peer 52411b8c21be44e3b21a18b16d8a98e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:00.044960 26432 raft_consensus.cc:1261] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 1 FOLLOWER]: Refusing update from remote peer 52411b8c21be44e3b21a18b16d8a98e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:00.046131 26771 consensus_queue.cc:1035] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb8c15695f96454e8b58d4f4da640347" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:00.046586 26767 consensus_queue.cc:1035] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd56d6ae628f402f9e6ca3f2f430a3b9" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:00.057018 26774 mvcc.cc:205] Tried to move back new op lower bound from 7298937815212621824 to 7298937815036456960. Current Snapshot: MvccSnapshot[applied={T|T < 7298937815212621824}]
I20260620 14:54:00.057346 26776 mvcc.cc:205] Tried to move back new op lower bound from 7298937815212621824 to 7298937815036456960. Current Snapshot: MvccSnapshot[applied={T|T < 7298937815212621824}]
I20260620 14:54:00.062611 26775 mvcc.cc:205] Tried to move back new op lower bound from 7298937815212621824 to 7298937815036456960. Current Snapshot: MvccSnapshot[applied={T|T < 7298937815212621824}]
I20260620 14:54:00.073441 26289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:00.073746 26289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:00.077626 26289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92cd833442d34970813dd4c02784b2ff: Sending DeleteTablet for 3 replicas of tablet 5d38c2c195a0445e9ca8f13a21271427
I20260620 14:54:00.078874 26412 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d38c2c195a0445e9ca8f13a21271427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:00 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:49666
I20260620 14:54:00.078998 26677 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d38c2c195a0445e9ca8f13a21271427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:00 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:34998
I20260620 14:54:00.079507 26545 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d38c2c195a0445e9ca8f13a21271427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:00 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:35236
I20260620 14:54:00.080487 26785 tablet_replica.cc:330] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: stopping tablet replica
I20260620 14:54:00.080595 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 26350
I20260620 14:54:00.080614 26783 tablet_replica.cc:330] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: stopping tablet replica
I20260620 14:54:00.081050 26783 raft_consensus.cc:2229] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:00.081082 26785 raft_consensus.cc:2229] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:00.081717 26785 raft_consensus.cc:2258] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:00.081765 26783 raft_consensus.cc:2258] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:00.084509 26785 ts_tablet_manager.cc:1905] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:00.087653 26783 ts_tablet_manager.cc:1905] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:00.091817 26784 tablet_replica.cc:330] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: stopping tablet replica
I20260620 14:54:00.092316 26784 raft_consensus.cc:2229] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:00.092854 26784 raft_consensus.cc:2258] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:00.093904 26785 ts_tablet_manager.cc:1918] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:00.094146 26785 log.cc:1196] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/5d38c2c195a0445e9ca8f13a21271427
I20260620 14:54:00.094743 26785 ts_tablet_manager.cc:1939] T 5d38c2c195a0445e9ca8f13a21271427 P eb8c15695f96454e8b58d4f4da640347: Deleting consensus metadata
I20260620 14:54:00.095240 26784 ts_tablet_manager.cc:1905] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:00.095834 26783 ts_tablet_manager.cc:1918] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:00.096103 26783 log.cc:1196] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/5d38c2c195a0445e9ca8f13a21271427
I20260620 14:54:00.096756 26274 catalog_manager.cc:4841] TS eb8c15695f96454e8b58d4f4da640347 (127.21.159.194:36327): tablet 5d38c2c195a0445e9ca8f13a21271427 (table test-table [id=8986f58a4ad24649840f02ca0623e801]) successfully deleted
I20260620 14:54:00.096941 26783 ts_tablet_manager.cc:1939] T 5d38c2c195a0445e9ca8f13a21271427 P 52411b8c21be44e3b21a18b16d8a98e0: Deleting consensus metadata
I20260620 14:54:00.098814 26273 catalog_manager.cc:4841] TS 52411b8c21be44e3b21a18b16d8a98e0 (127.21.159.195:46211): tablet 5d38c2c195a0445e9ca8f13a21271427 (table test-table [id=8986f58a4ad24649840f02ca0623e801]) successfully deleted
I20260620 14:54:00.102223 26784 ts_tablet_manager.cc:1918] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:00.102473 26784 log.cc:1196] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/5d38c2c195a0445e9ca8f13a21271427
I20260620 14:54:00.103104 26784 ts_tablet_manager.cc:1939] T 5d38c2c195a0445e9ca8f13a21271427 P cd56d6ae628f402f9e6ca3f2f430a3b9: Deleting consensus metadata
I20260620 14:54:00.105079 26273 catalog_manager.cc:4841] TS cd56d6ae628f402f9e6ca3f2f430a3b9 (127.21.159.193:40215): tablet 5d38c2c195a0445e9ca8f13a21271427 (table test-table [id=8986f58a4ad24649840f02ca0623e801]) successfully deleted
Jun 20 14:54:00 dist-test-slave-3mqx krb5kdc[26245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:54:00.118515 26756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:00.082926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.21.159.193:40215)
0620 14:54:00.083167 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:00.083176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:00.086383 (+ 3207us) server_negotiation.cc:407] Connection header received
0620 14:54:00.098902 (+ 12519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:00.098909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:00.098999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:00.099107 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:00.100993 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.102191 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.104163 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.104854 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.104963 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:00.110392 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:00.110414 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:00.110437 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:00.110489 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:00.114612 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:00.115752 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:00.115763 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:00.115767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:00.115954 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:00.116598 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:00.116604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:00.116608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:00.117122 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:00.117473 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:00.117669 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:54:00.117910 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:54:00.119343 26472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:00.248447 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 26350
I20260620 14:54:00.266610 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 26483
Jun 20 14:54:00 dist-test-slave-3mqx krb5kdc[26245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:54:00.284549 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:00.267517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.21.159.194:36327)
0620 14:54:00.267701 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:00.267709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:00.267781 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:54:00.267974 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:00.267979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:00.268072 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:00.268197 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:00.269813 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.270564 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.272481 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.272926 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.273031 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:00.278294 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:00.278310 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:00.278328 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:00.278368 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:00.281495 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:00.282262 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:00.282270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:00.282272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:00.282406 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:00.282795 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:00.282800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:00.282804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:00.283269 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:00.283566 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:00.283916 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:54:00.284135 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:54:00.285341 26605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:00.376561 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 26483
I20260620 14:54:00.397589 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 26615
Jun 20 14:54:00 dist-test-slave-3mqx krb5kdc[26245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:54:00.416508 26760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:00.398893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.21.159.195:46211)
0620 14:54:00.399086 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:00.399092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:00.399114 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:54:00.399282 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:00.399288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:00.399413 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:00.399539 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:00.400945 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.401751 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.403008 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:00.403558 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:00.403736 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:00.408313 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:00.408332 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:00.408366 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:00.408437 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:00.412928 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:00.413864 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:00.413899 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:00.413903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:00.414087 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:00.414569 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:00.414575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:00.414578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:00.415205 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:00.415565 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:00.415776 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:54:00.416009 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:54:00.417500 26737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:00.539956 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 26615
I20260620 14:54:00.559546 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 26258
I20260620 14:54:00.560787 26319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:00.669303 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 26258
2026-06-20T14:54:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4487 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:00Z Disabled control of system clock
I20260620 14:54:00.737872 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33185
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:54:01.079430 26794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:01.079813 26794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:01.079996 26794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:01.087630 26794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:01.087793 26794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:01.087882 26794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:01.087997 26794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:01.088076 26794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:01.097841 26794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:01.099308 26794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:01.101166 26794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:01.114630 26802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:01.114588 26800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:01.114675 26799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:01.119704 26794 server_base.cc:1034] running on GCE node
I20260620 14:54:01.120579 26794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:01.122143 26794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:01.123356 26794 hybrid_clock.cc:648] HybridClock initialized: now 1781967241123311 us; error 51 us; skew 500 ppm
I20260620 14:54:01.123703 26794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:01.126129 26794 webserver.cc:460] Webserver started at http://127.21.159.254:33135/ using document root <none> and password file <none>
I20260620 14:54:01.126857 26794 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:01.126950 26794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:01.127350 26794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:01.130430 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "424e4c01dce4464eab8a70d65ebdb10d"
format_stamp: "Formatted at 2026-06-20 14:54:01 on dist-test-slave-3mqx"
server_key: "a9aa74775daebedafb5b714fc78b7d55"
server_key_iv: "b174748f9de1f04f6433bb2b3967d753"
server_key_version: "encryptionkey@0"
I20260620 14:54:01.131304 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "424e4c01dce4464eab8a70d65ebdb10d"
format_stamp: "Formatted at 2026-06-20 14:54:01 on dist-test-slave-3mqx"
server_key: "a9aa74775daebedafb5b714fc78b7d55"
server_key_iv: "b174748f9de1f04f6433bb2b3967d753"
server_key_version: "encryptionkey@0"
I20260620 14:54:01.138947 26794 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 14:54:01.144959 26808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:01.147339 26794 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:54:01.147570 26794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "424e4c01dce4464eab8a70d65ebdb10d"
format_stamp: "Formatted at 2026-06-20 14:54:01 on dist-test-slave-3mqx"
server_key: "a9aa74775daebedafb5b714fc78b7d55"
server_key_iv: "b174748f9de1f04f6433bb2b3967d753"
server_key_version: "encryptionkey@0"
I20260620 14:54:01.147866 26794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:01.156668 26794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:01.157853 26794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:01.158227 26794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:01.181631 26794 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:33185
I20260620 14:54:01.181731 26859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:33185 every 8 connection(s)
I20260620 14:54:01.183208 26794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:01.188869 26860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:01.189489 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 26794
I20260620 14:54:01.189694 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:01.190348 22143 external_mini_cluster.cc:1428] Setting key 83805e5d778494f0d1715b65eda1577f
I20260620 14:54:01.201109 26863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:01.193144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.21.159.254:33185)
0620 14:54:01.194479 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:01.194495 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:01.194554 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:54:01.196702 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:01.196752 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:01.197298 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:01.197704 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:01.199139 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:01.199156 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:01.199179 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:01.199205 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:01.199281 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:01.199314 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:01.199527 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:01.199660 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:54:01.200010 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":495,"threads_started":1}
I20260620 14:54:01.204761 26860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d: Bootstrap starting.
I20260620 14:54:01.208109 26860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:01.209292 26860 log.cc:824] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:01.212350 26860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d: No bootstrap required, opened a new log
I20260620 14:54:01.219754 26860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e4c01dce4464eab8a70d65ebdb10d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33185 } }
I20260620 14:54:01.220177 26860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:01.220309 26860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424e4c01dce4464eab8a70d65ebdb10d, State: Initialized, Role: FOLLOWER
I20260620 14:54:01.220914 26860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e4c01dce4464eab8a70d65ebdb10d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33185 } }
I20260620 14:54:01.221122 26860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:01.221269 26860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:01.221459 26860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:01.223805 26860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e4c01dce4464eab8a70d65ebdb10d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33185 } }
I20260620 14:54:01.224359 26860 leader_election.cc:302] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 424e4c01dce4464eab8a70d65ebdb10d; no voters:
I20260620 14:54:01.224977 26860 leader_election.cc:288] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:01.225170 26865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:01.225914 26865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [term 1 LEADER]: Becoming Leader. State: Replica: 424e4c01dce4464eab8a70d65ebdb10d, State: Running, Role: LEADER
I20260620 14:54:01.226485 26865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e4c01dce4464eab8a70d65ebdb10d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33185 } }
I20260620 14:54:01.227334 26860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:01.229727 26867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 424e4c01dce4464eab8a70d65ebdb10d. Latest consensus state: current_term: 1 leader_uuid: "424e4c01dce4464eab8a70d65ebdb10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e4c01dce4464eab8a70d65ebdb10d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33185 } } }
I20260620 14:54:01.229674 26866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "424e4c01dce4464eab8a70d65ebdb10d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424e4c01dce4464eab8a70d65ebdb10d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33185 } } }
I20260620 14:54:01.230252 26866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:01.230261 26867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:01.234139 26873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:01.239861 26873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:01.250483 26873 catalog_manager.cc:1347] Generated new cluster ID: 2132c2dbb2a64c6781d07c37e8e60489
I20260620 14:54:01.250655 26873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:01.280362 26873 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:01.280881 26873 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:01.293684 26873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d: Generated new TSK 0
I20260620 14:54:01.294701 26873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:01.304812 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:33185
--builtin_ntp_servers=127.21.159.212:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:54:01.660203 26884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:01.660527 26884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:01.660617 26884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:01.668426 26884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:01.668548 26884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:01.668663 26884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:01.678170 26884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:33185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:01.679503 26884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:01.681360 26884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:01.695616 26889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:01.695616 26890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:01.695886 26892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:01.696506 26884 server_base.cc:1034] running on GCE node
I20260620 14:54:01.697238 26884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:01.698411 26884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:01.699694 26884 hybrid_clock.cc:648] HybridClock initialized: now 1781967241699633 us; error 45 us; skew 500 ppm
I20260620 14:54:01.700101 26884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:01.702708 26884 webserver.cc:460] Webserver started at http://127.21.159.193:42561/ using document root <none> and password file <none>
I20260620 14:54:01.703644 26884 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:01.703784 26884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:01.704141 26884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:01.707535 26884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "43ed90d0c62b4b10a2541a0f5f2317b1"
format_stamp: "Formatted at 2026-06-20 14:54:01 on dist-test-slave-3mqx"
server_key: "9c566e220d056eb9616ee34b7ac6939f"
server_key_iv: "aef127d5bca82c5f1ca7c8f7574ed677"
server_key_version: "encryptionkey@0"
I20260620 14:54:01.708489 26884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "43ed90d0c62b4b10a2541a0f5f2317b1"
format_stamp: "Formatted at 2026-06-20 14:54:01 on dist-test-slave-3mqx"
server_key: "9c566e220d056eb9616ee34b7ac6939f"
server_key_iv: "aef127d5bca82c5f1ca7c8f7574ed677"
server_key_version: "encryptionkey@0"
I20260620 14:54:01.714660 26884 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 14:54:01.718937 26898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:01.720788 26884 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:54:01.721046 26884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "43ed90d0c62b4b10a2541a0f5f2317b1"
format_stamp: "Formatted at 2026-06-20 14:54:01 on dist-test-slave-3mqx"
server_key: "9c566e220d056eb9616ee34b7ac6939f"
server_key_iv: "aef127d5bca82c5f1ca7c8f7574ed677"
server_key_version: "encryptionkey@0"
I20260620 14:54:01.721324 26884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:01.727676 26884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:01.728677 26884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:01.729020 26884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:01.730067 26884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:01.731788 26884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:01.731927 26884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:01.732053 26884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:01.732137 26884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:01.767434 26884 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:33495
I20260620 14:54:01.767499 27010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:33495 every 8 connection(s)
I20260620 14:54:01.768836 26884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:01.777177 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 26884
I20260620 14:54:01.777416 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:01.777946 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:01.771662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:34951 (local address 127.21.159.254:33185)
0620 14:54:01.772135 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:01.772140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:01.773521 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 14:54:01.775371 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:01.775377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:01.775455 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:01.775577 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:01.777068 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:01.777073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:01.777077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:01.777095 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:01.777144 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:01.777157 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:01.777436 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:01.777646 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:54:01.777702 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":231,"threads_started":1}
I20260620 14:54:01.778239 22143 external_mini_cluster.cc:1428] Setting key b67c4408272f44934b44c96150ecb9b5
I20260620 14:54:01.779510 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:01.772303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33185 (local address 127.21.159.193:34951)
0620 14:54:01.773268 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:01.773372 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:01.774971 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:01.775844 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:01.775852 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:01.775954 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:01.776628 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:01.776631 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:01.776704 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:01.776831 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:01.777394 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:01.777408 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:01.777417 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:54:01.777735 (+ 318us) client_negotiation.cc:240] Negotiation successful
0620 14:54:01.778045 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":287,"threads_started":1}
I20260620 14:54:01.781559 27011 heartbeater.cc:347] Connected to a master server at 127.21.159.254:33185
I20260620 14:54:01.781958 27011 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:01.782068 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:33185
--builtin_ntp_servers=127.21.159.212:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:54:01.782824 27011 heartbeater.cc:511] Master 127.21.159.254:33185 requested a full tablet report, sending...
I20260620 14:54:01.785354 26825 ts_manager.cc:194] Registered new tserver with Master: 43ed90d0c62b4b10a2541a0f5f2317b1 (127.21.159.193:33495)
W20260620 14:54:02.137372 27015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:02.137728 27015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:02.137889 27015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:02.145550 27015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:02.145721 27015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:02.145921 27015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:02.155866 27015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:33185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:02.157275 27015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:02.159217 27015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:02.173687 27020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:02.174017 27021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:02.174021 27023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:02.175453 27015 server_base.cc:1034] running on GCE node
I20260620 14:54:02.176180 27015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:02.177316 27015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:02.178586 27015 hybrid_clock.cc:648] HybridClock initialized: now 1781967242178502 us; error 76 us; skew 500 ppm
I20260620 14:54:02.178968 27015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:02.181511 27015 webserver.cc:460] Webserver started at http://127.21.159.194:39047/ using document root <none> and password file <none>
I20260620 14:54:02.182278 27015 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:02.182423 27015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:02.182767 27015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:02.185860 27015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "a9189f56544946ba9434f78e2e4a8af6"
format_stamp: "Formatted at 2026-06-20 14:54:02 on dist-test-slave-3mqx"
server_key: "acc6e50d38b6b9b0b30f4160bff3db05"
server_key_iv: "5c794e87026b64ae7ee9c25564d31f8e"
server_key_version: "encryptionkey@0"
I20260620 14:54:02.186757 27015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "a9189f56544946ba9434f78e2e4a8af6"
format_stamp: "Formatted at 2026-06-20 14:54:02 on dist-test-slave-3mqx"
server_key: "acc6e50d38b6b9b0b30f4160bff3db05"
server_key_iv: "5c794e87026b64ae7ee9c25564d31f8e"
server_key_version: "encryptionkey@0"
I20260620 14:54:02.192551 27015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 14:54:02.196363 27029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:02.198010 27015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:54:02.198236 27015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "a9189f56544946ba9434f78e2e4a8af6"
format_stamp: "Formatted at 2026-06-20 14:54:02 on dist-test-slave-3mqx"
server_key: "acc6e50d38b6b9b0b30f4160bff3db05"
server_key_iv: "5c794e87026b64ae7ee9c25564d31f8e"
server_key_version: "encryptionkey@0"
I20260620 14:54:02.198568 27015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:02.204638 27015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:02.205478 27015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:02.205710 27015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:02.206604 27015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:02.208086 27015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:02.208181 27015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:02.208257 27015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:02.208313 27015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:02.242736 27015 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:36551
I20260620 14:54:02.242787 27141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:36551 every 8 connection(s)
I20260620 14:54:02.244148 27015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:02.244665 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27015
I20260620 14:54:02.244868 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:02.245541 22143 external_mini_cluster.cc:1428] Setting key 86eccf27129c939a99256b4a95d9f12f
I20260620 14:54:02.249037 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:33185
--builtin_ntp_servers=127.21.159.212:36961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:54:02.253433 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.247197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:45493 (local address 127.21.159.254:33185)
0620 14:54:02.247388 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:02.247395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:02.249158 (+ 1763us) server_negotiation.cc:407] Connection header received
0620 14:54:02.250944 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:02.250950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:02.251035 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:02.251209 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:02.252658 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:02.252665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:02.252670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:02.252691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:02.252732 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:02.252745 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:02.252875 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:02.253086 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:54:02.253160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:54:02.254556 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.247959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33185 (local address 127.21.159.194:45493)
0620 14:54:02.248985 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:02.249031 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:02.250641 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:02.251473 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:02.251483 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:02.251590 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:02.252328 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:02.252331 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:02.252374 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:02.252470 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:02.252841 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:02.252852 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:02.252860 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:54:02.253163 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 14:54:02.253465 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":301,"threads_started":1}
I20260620 14:54:02.256040 27142 heartbeater.cc:347] Connected to a master server at 127.21.159.254:33185
I20260620 14:54:02.256364 27142 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:02.257102 27142 heartbeater.cc:511] Master 127.21.159.254:33185 requested a full tablet report, sending...
I20260620 14:54:02.258677 26825 ts_manager.cc:194] Registered new tserver with Master: a9189f56544946ba9434f78e2e4a8af6 (127.21.159.194:36551)
W20260620 14:54:02.591981 27145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:02.592307 27145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:02.592473 27145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:02.599977 27145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:02.600106 27145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:02.600310 27145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:02.609791 27145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:33185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:02.611316 27145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:02.613229 27145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:02.627374 27150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:02.627341 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:02.627421 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:02.632768 27145 server_base.cc:1034] running on GCE node
I20260620 14:54:02.633499 27145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:02.634519 27145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:02.635756 27145 hybrid_clock.cc:648] HybridClock initialized: now 1781967242635710 us; error 97 us; skew 500 ppm
I20260620 14:54:02.636139 27145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:02.638562 27145 webserver.cc:460] Webserver started at http://127.21.159.195:45285/ using document root <none> and password file <none>
I20260620 14:54:02.639402 27145 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:02.639496 27145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:02.639794 27145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:02.642879 27145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "8059ac4f2cb24b89a185be49b78e8a36"
format_stamp: "Formatted at 2026-06-20 14:54:02 on dist-test-slave-3mqx"
server_key: "9b04170bd1af2af8e61d2fe569d1c7ea"
server_key_iv: "92f8b9863cb630f29411d114e83cc7c7"
server_key_version: "encryptionkey@0"
I20260620 14:54:02.643821 27145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "8059ac4f2cb24b89a185be49b78e8a36"
format_stamp: "Formatted at 2026-06-20 14:54:02 on dist-test-slave-3mqx"
server_key: "9b04170bd1af2af8e61d2fe569d1c7ea"
server_key_iv: "92f8b9863cb630f29411d114e83cc7c7"
server_key_version: "encryptionkey@0"
I20260620 14:54:02.649550 27145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:54:02.653766 27159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:02.655488 27145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:02.655719 27145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "8059ac4f2cb24b89a185be49b78e8a36"
format_stamp: "Formatted at 2026-06-20 14:54:02 on dist-test-slave-3mqx"
server_key: "9b04170bd1af2af8e61d2fe569d1c7ea"
server_key_iv: "92f8b9863cb630f29411d114e83cc7c7"
server_key_version: "encryptionkey@0"
I20260620 14:54:02.655990 27145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:02.662294 27145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:02.663282 27145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:02.663592 27145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:02.664570 27145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:02.666252 27145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:02.666380 27145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:02.666525 27145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:02.666615 27145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:02.701632 27145 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:41301
I20260620 14:54:02.701685 27271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:41301 every 8 connection(s)
I20260620 14:54:02.703276 27145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:02.711510 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27145
I20260620 14:54:02.711709 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:02.712304 22143 external_mini_cluster.cc:1428] Setting key b12e3d21fb8500d2cc3705cf43fbedc0
I20260620 14:54:02.712584 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.706239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:47461 (local address 127.21.159.254:33185)
0620 14:54:02.706448 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:02.706453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:02.707990 (+ 1537us) server_negotiation.cc:407] Connection header received
0620 14:54:02.709804 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:02.709810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:02.709899 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:02.710027 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:02.711713 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:02.711725 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:02.711732 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:02.711755 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:02.711803 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:02.711822 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:02.711949 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:02.712198 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:54:02.712251 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:54:02.713914 27274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.706786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33185 (local address 127.21.159.195:47461)
0620 14:54:02.707813 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:02.707852 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:02.709491 (+ 1639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:02.710292 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:02.710300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:02.710407 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:02.711221 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:02.711226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:02.711316 (+ 90us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:02.711445 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:02.711947 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:02.711959 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:02.711968 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:54:02.712284 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 14:54:02.712603 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":364,"threads_started":1}
I20260620 14:54:02.715795 27272 heartbeater.cc:347] Connected to a master server at 127.21.159.254:33185
I20260620 14:54:02.716092 27272 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:02.716709 27272 heartbeater.cc:511] Master 127.21.159.254:33185 requested a full tablet report, sending...
I20260620 14:54:02.718163 26825 ts_manager.cc:194] Registered new tserver with Master: 8059ac4f2cb24b89a185be49b78e8a36 (127.21.159.195:41301)
I20260620 14:54:02.728878 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:54:02.737541 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.733966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.21.159.254:33185)
0620 14:54:02.734220 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:02.734231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:02.734784 (+ 553us) server_negotiation.cc:407] Connection header received
0620 14:54:02.735132 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:02.735137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:02.735197 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:02.735295 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:02.736715 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:02.736721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:02.736725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:02.736748 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:02.736792 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:02.736808 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:02.736953 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:02.737056 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:54:02.737119 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:54:02.742385 26825 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:02.749488 26825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:02.784804 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.762959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:41301 (local address 127.0.0.1:40122)
0620 14:54:02.771993 (+ 9034us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:02.772058 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:02.772248 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:02.781612 (+ 9364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:02.781625 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:02.781782 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:02.783149 (+ 1367us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:02.783151 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:02.783180 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:02.783246 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:02.783922 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:02.783955 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:02.783967 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:54:02.784157 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 14:54:02.784258 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8662,"thread_start_us":19846,"threads_started":1}
I20260620 14:54:02.784840 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.763178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36202)
0620 14:54:02.780636 (+ 17458us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:02.780660 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:02.780816 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:02.782507 (+ 1691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:02.782516 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:02.782552 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:02.783516 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:02.783519 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:02.783534 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:02.783570 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:02.784075 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:02.784082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:02.784087 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:02.784235 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 14:54:02.784304 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17275,"thread_start_us":17116,"threads_started":1}
I20260620 14:54:02.785475 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.763769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.21.159.193:33495)
0620 14:54:02.780252 (+ 16483us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:02.780265 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:02.780962 (+ 697us) server_negotiation.cc:407] Connection header received
0620 14:54:02.781048 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:02.781059 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:02.781331 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:02.781509 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:02.783730 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:02.783739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:02.783753 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:02.783773 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:02.783838 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:02.783859 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:02.784018 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:02.784942 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 14:54:02.785059 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16249,"thread_start_us":16045,"threads_started":1}
I20260620 14:54:02.785778 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.763167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.21.159.195:41301)
0620 14:54:02.781030 (+ 17863us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:02.781038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:02.781070 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:54:02.781134 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:02.781140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:02.781331 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:02.781514 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:02.783498 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:02.783508 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:02.783532 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:02.783548 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:02.783641 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:02.783665 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:02.783863 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:02.785172 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0620 14:54:02.785310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17685,"thread_start_us":17526,"threads_started":1}
I20260620 14:54:02.786876 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.766067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:36551 (local address 127.0.0.1:50822)
0620 14:54:02.783192 (+ 17125us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:02.783216 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:02.783352 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:02.784980 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:02.784984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:02.785005 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:02.785751 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:02.785753 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:02.785766 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:02.785807 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:02.786374 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:02.786379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:02.786388 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 14:54:02.786521 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 14:54:02.786562 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16940,"thread_start_us":5343,"threads_started":1}
I20260620 14:54:02.786964 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:02.766202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.21.159.194:36551)
0620 14:54:02.776590 (+ 10388us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:02.776604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:02.783369 (+ 6765us) server_negotiation.cc:407] Connection header received
0620 14:54:02.783525 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:02.783534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:02.783823 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:02.783979 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:02.786024 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:02.786034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:02.786051 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:02.786071 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:02.786134 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:02.786149 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:02.786303 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:02.786546 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:54:02.786656 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10172,"thread_start_us":10032,"threads_started":1}
I20260620 14:54:02.788612 27011 heartbeater.cc:503] Master 127.21.159.254:33185 was elected leader, sending a full tablet report...
I20260620 14:54:02.790592 27207 tablet_service.cc:1467] Processing CreateTablet for tablet 32b15a350a4443f5b45bcbe939504868 (DEFAULT_TABLE table=test-table [id=6bcb9989509b4cb1affeba5d64ad59bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:02.790592 27077 tablet_service.cc:1467] Processing CreateTablet for tablet 32b15a350a4443f5b45bcbe939504868 (DEFAULT_TABLE table=test-table [id=6bcb9989509b4cb1affeba5d64ad59bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:02.790802 26946 tablet_service.cc:1467] Processing CreateTablet for tablet 32b15a350a4443f5b45bcbe939504868 (DEFAULT_TABLE table=test-table [id=6bcb9989509b4cb1affeba5d64ad59bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:02.792165 26946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b15a350a4443f5b45bcbe939504868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:02.792193 27207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b15a350a4443f5b45bcbe939504868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:02.792193 27077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b15a350a4443f5b45bcbe939504868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:02.804410 27290 tablet_bootstrap.cc:492] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: Bootstrap starting.
I20260620 14:54:02.808254 27290 tablet_bootstrap.cc:654] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:02.809224 27291 tablet_bootstrap.cc:492] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Bootstrap starting.
I20260620 14:54:02.810019 27290 log.cc:824] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:02.811558 27292 tablet_bootstrap.cc:492] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Bootstrap starting.
I20260620 14:54:02.814093 27290 tablet_bootstrap.cc:492] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: No bootstrap required, opened a new log
I20260620 14:54:02.814435 27290 ts_tablet_manager.cc:1397] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 14:54:02.815805 27292 tablet_bootstrap.cc:654] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:02.817286 27291 tablet_bootstrap.cc:654] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:02.817224 27292 log.cc:824] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:02.818864 27291 log.cc:824] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:02.822830 27292 tablet_bootstrap.cc:492] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: No bootstrap required, opened a new log
I20260620 14:54:02.823096 27291 tablet_bootstrap.cc:492] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: No bootstrap required, opened a new log
I20260620 14:54:02.823382 27291 ts_tablet_manager.cc:1397] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260620 14:54:02.823364 27292 ts_tablet_manager.cc:1397] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 14:54:02.824376 27290 raft_consensus.cc:348] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:02.824805 27290 raft_consensus.cc:374] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:02.824906 27290 raft_consensus.cc:729] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ed90d0c62b4b10a2541a0f5f2317b1, State: Initialized, Role: FOLLOWER
I20260620 14:54:02.825677 27290 consensus_queue.cc:260] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:02.827989 27290 ts_tablet_manager.cc:1428] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260620 14:54:02.834758 27292 raft_consensus.cc:348] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:02.834758 27291 raft_consensus.cc:348] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:02.835369 27291 raft_consensus.cc:374] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:02.835371 27292 raft_consensus.cc:374] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:02.835487 27291 raft_consensus.cc:729] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8059ac4f2cb24b89a185be49b78e8a36, State: Initialized, Role: FOLLOWER
I20260620 14:54:02.835487 27292 raft_consensus.cc:729] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9189f56544946ba9434f78e2e4a8af6, State: Initialized, Role: FOLLOWER
I20260620 14:54:02.836237 27291 consensus_queue.cc:260] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:02.836238 27292 consensus_queue.cc:260] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:02.837488 27142 heartbeater.cc:503] Master 127.21.159.254:33185 was elected leader, sending a full tablet report...
I20260620 14:54:02.837957 27272 heartbeater.cc:503] Master 127.21.159.254:33185 was elected leader, sending a full tablet report...
I20260620 14:54:02.838387 27292 ts_tablet_manager.cc:1428] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 14:54:02.838536 27291 ts_tablet_manager.cc:1428] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
W20260620 14:54:02.955197 27273 tablet.cc:2369] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:54:02.996618 27143 tablet.cc:2369] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:54:03.021701 27012 tablet.cc:2369] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:03.034098 27297 raft_consensus.cc:482] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:03.034482 27297 raft_consensus.cc:504] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:03.036803 27297 leader_election.cc:288] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8059ac4f2cb24b89a185be49b78e8a36 (127.21.159.195:41301), 43ed90d0c62b4b10a2541a0f5f2317b1 (127.21.159.193:33495)
I20260620 14:54:03.046140 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.037353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:42429 (local address 127.21.159.193:33495)
0620 14:54:03.037569 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:03.037578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:03.038543 (+ 965us) server_negotiation.cc:407] Connection header received
0620 14:54:03.038704 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:03.038708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:03.038760 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:03.038880 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:03.045471 (+ 6591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:03.045475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:03.045478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:03.045497 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:03.045530 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:03.045542 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:03.045695 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:03.045798 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:54:03.045863 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 14:54:03.046175 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.037223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:56261 (local address 127.21.159.195:41301)
0620 14:54:03.037432 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:03.037440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:03.038424 (+ 984us) server_negotiation.cc:407] Connection header received
0620 14:54:03.038679 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:03.038685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:03.038767 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:03.038883 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:03.045457 (+ 6574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:03.045462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:03.045466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:03.045488 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:03.045521 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:03.045534 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:03.045686 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:03.045752 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:54:03.045827 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:54:03.046483 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.037377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:41301 (local address 127.21.159.194:56261)
0620 14:54:03.038283 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.038306 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:03.038491 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:03.038940 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:03.038945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:03.038971 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:03.045142 (+ 6171us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:03.045146 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:03.045165 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:03.045227 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:03.045644 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:03.045649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:03.045653 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:03.045879 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 14:54:03.045997 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":49,"thread_start_us":513,"threads_started":1}
I20260620 14:54:03.046945 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.037501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.21.159.194:42429)
0620 14:54:03.038425 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.038446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:03.038547 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:03.038902 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:03.038951 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:03.038976 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:03.045185 (+ 6209us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:03.045188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:03.045203 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:03.045246 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:03.045672 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:03.045676 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:03.045681 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:03.045873 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 14:54:03.045950 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":500,"threads_started":1}
I20260620 14:54:03.047991 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b15a350a4443f5b45bcbe939504868" candidate_uuid: "a9189f56544946ba9434f78e2e4a8af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8059ac4f2cb24b89a185be49b78e8a36" is_pre_election: true
I20260620 14:54:03.048049 26966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b15a350a4443f5b45bcbe939504868" candidate_uuid: "a9189f56544946ba9434f78e2e4a8af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" is_pre_election: true
I20260620 14:54:03.048617 26966 raft_consensus.cc:2454] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9189f56544946ba9434f78e2e4a8af6 in term 0.
I20260620 14:54:03.048617 27227 raft_consensus.cc:2454] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9189f56544946ba9434f78e2e4a8af6 in term 0.
I20260620 14:54:03.049407 27031 leader_election.cc:302] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ed90d0c62b4b10a2541a0f5f2317b1, a9189f56544946ba9434f78e2e4a8af6; no voters:
I20260620 14:54:03.049875 27297 raft_consensus.cc:2781] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:03.050033 27297 raft_consensus.cc:482] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:03.050139 27297 raft_consensus.cc:3037] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:03.052448 27297 raft_consensus.cc:504] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:03.053095 27297 leader_election.cc:288] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [CANDIDATE]: Term 1 election: Requested vote from peers 8059ac4f2cb24b89a185be49b78e8a36 (127.21.159.195:41301), 43ed90d0c62b4b10a2541a0f5f2317b1 (127.21.159.193:33495)
I20260620 14:54:03.053727 27227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b15a350a4443f5b45bcbe939504868" candidate_uuid: "a9189f56544946ba9434f78e2e4a8af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8059ac4f2cb24b89a185be49b78e8a36"
I20260620 14:54:03.053716 26966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32b15a350a4443f5b45bcbe939504868" candidate_uuid: "a9189f56544946ba9434f78e2e4a8af6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1"
I20260620 14:54:03.053944 27227 raft_consensus.cc:3037] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:03.054071 26966 raft_consensus.cc:3037] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:03.056445 27227 raft_consensus.cc:2454] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9189f56544946ba9434f78e2e4a8af6 in term 1.
I20260620 14:54:03.056974 27033 leader_election.cc:302] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8059ac4f2cb24b89a185be49b78e8a36, a9189f56544946ba9434f78e2e4a8af6; no voters:
I20260620 14:54:03.056980 26966 raft_consensus.cc:2454] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9189f56544946ba9434f78e2e4a8af6 in term 1.
I20260620 14:54:03.057328 27297 raft_consensus.cc:2781] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:03.057858 27297 raft_consensus.cc:686] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 1 LEADER]: Becoming Leader. State: Replica: a9189f56544946ba9434f78e2e4a8af6, State: Running, Role: LEADER
I20260620 14:54:03.058465 27297 consensus_queue.cc:237] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } }
I20260620 14:54:03.063591 26825 catalog_manager.cc:5495] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9189f56544946ba9434f78e2e4a8af6 (127.21.159.194). New cstate: current_term: 1 leader_uuid: "a9189f56544946ba9434f78e2e4a8af6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9189f56544946ba9434f78e2e4a8af6" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 36551 } health_report { overall_health: HEALTHY } } }
I20260620 14:54:03.145576 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.143477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.21.159.194:36551)
0620 14:54:03.143656 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:03.143660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:03.143777 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:54:03.143964 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:03.143968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:03.144051 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:03.144137 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:03.145036 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:03.145040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:03.145044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:03.145059 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:03.145090 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:03.145102 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:03.145193 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:03.145302 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:54:03.145346 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:54:03.154721 27227 raft_consensus.cc:1261] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 1 FOLLOWER]: Refusing update from remote peer a9189f56544946ba9434f78e2e4a8af6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:03.154819 26966 raft_consensus.cc:1261] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 1 FOLLOWER]: Refusing update from remote peer a9189f56544946ba9434f78e2e4a8af6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:03.156073 27297 consensus_queue.cc:1035] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43ed90d0c62b4b10a2541a0f5f2317b1" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:03.156464 27301 consensus_queue.cc:1035] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8059ac4f2cb24b89a185be49b78e8a36" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:03.171288 27304 mvcc.cc:205] Tried to move back new op lower bound from 7298937827950047232 to 7298937827571445760. Current Snapshot: MvccSnapshot[applied={T|T < 7298937827950047232}]
I20260620 14:54:03.188129 26825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:03.188477 26825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:03.193061 26825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 424e4c01dce4464eab8a70d65ebdb10d: Sending DeleteTablet for 3 replicas of tablet 32b15a350a4443f5b45bcbe939504868
I20260620 14:54:03.194367 27207 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b15a350a4443f5b45bcbe939504868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:03 UTC) from {username='slave'} at 127.0.0.1:40122
I20260620 14:54:03.194471 26946 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b15a350a4443f5b45bcbe939504868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:03 UTC) from {username='slave'} at 127.0.0.1:36202
I20260620 14:54:03.195847 27077 tablet_service.cc:1514] Processing DeleteTablet for tablet 32b15a350a4443f5b45bcbe939504868 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:03 UTC) from {username='slave'} at 127.0.0.1:50822
I20260620 14:54:03.202971 27316 tablet_replica.cc:330] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: stopping tablet replica
I20260620 14:54:03.203601 27316 raft_consensus.cc:2229] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:03.204419 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 26884
I20260620 14:54:03.204797 27315 tablet_replica.cc:330] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: stopping tablet replica
I20260620 14:54:03.205338 27315 raft_consensus.cc:2229] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:03.205869 27315 raft_consensus.cc:2258] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:03.206772 27316 raft_consensus.cc:2258] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:03.208498 27314 tablet_replica.cc:330] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: stopping tablet replica
I20260620 14:54:03.208967 27314 raft_consensus.cc:2229] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:03.209561 27314 raft_consensus.cc:2258] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:03.210405 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.205690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.21.159.193:33495)
0620 14:54:03.205891 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:03.205898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:03.207690 (+ 1792us) server_negotiation.cc:407] Connection header received
0620 14:54:03.207926 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:03.207932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:03.208054 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:03.208176 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:03.209456 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:03.209462 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:03.209467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:03.209492 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:03.209559 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:03.209580 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:03.209702 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:03.209988 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 14:54:03.210077 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:54:03.210629 27316 ts_tablet_manager.cc:1905] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:03.211592 27006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:03.214196 27314 ts_tablet_manager.cc:1905] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:03.221774 27316 ts_tablet_manager.cc:1918] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:03.221999 27316 log.cc:1196] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/32b15a350a4443f5b45bcbe939504868
I20260620 14:54:03.222484 27314 ts_tablet_manager.cc:1918] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:03.222647 27316 ts_tablet_manager.cc:1939] T 32b15a350a4443f5b45bcbe939504868 P a9189f56544946ba9434f78e2e4a8af6: Deleting consensus metadata
I20260620 14:54:03.222808 27314 log.cc:1196] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/32b15a350a4443f5b45bcbe939504868
I20260620 14:54:03.223392 27314 ts_tablet_manager.cc:1939] T 32b15a350a4443f5b45bcbe939504868 P 8059ac4f2cb24b89a185be49b78e8a36: Deleting consensus metadata
I20260620 14:54:03.224828 26809 catalog_manager.cc:4841] TS a9189f56544946ba9434f78e2e4a8af6 (127.21.159.194:36551): tablet 32b15a350a4443f5b45bcbe939504868 (table test-table [id=6bcb9989509b4cb1affeba5d64ad59bb]) successfully deleted
I20260620 14:54:03.225355 26812 catalog_manager.cc:4841] TS 8059ac4f2cb24b89a185be49b78e8a36 (127.21.159.195:41301): tablet 32b15a350a4443f5b45bcbe939504868 (table test-table [id=6bcb9989509b4cb1affeba5d64ad59bb]) successfully deleted
I20260620 14:54:03.212103 27315 ts_tablet_manager.cc:1905] T 32b15a350a4443f5b45bcbe939504868 P 43ed90d0c62b4b10a2541a0f5f2317b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:03.310876 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 26884
W20260620 14:54:03.328454 26810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.159.193:33495 (error 108)
I20260620 14:54:03.329319 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27015
I20260620 14:54:03.330662 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.329792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36216)
0620 14:54:03.330254 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.330391 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":247,"threads_started":1}
W20260620 14:54:03.331559 26810 catalog_manager.cc:4584] TS 43ed90d0c62b4b10a2541a0f5f2317b1 (127.21.159.193:33495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32b15a350a4443f5b45bcbe939504868: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
I20260620 14:54:03.332918 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.330747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.21.159.194:36551)
0620 14:54:03.330931 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:03.330935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:03.330956 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:54:03.331033 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:03.331038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:03.331157 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:03.331268 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:03.332257 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:03.332263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:03.332268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:03.332285 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:03.332324 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:03.332336 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:03.332473 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:03.332588 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:54:03.332645 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:54:03.333600 27137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:03.385421 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.384909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36218)
0620 14:54:03.385152 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.385281 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 14:54:03.386682 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.386261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36224)
0620 14:54:03.386428 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.386517 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:54:03.442751 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27015
I20260620 14:54:03.447109 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.446619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36230)
0620 14:54:03.446822 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.446919 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:54:03.448348 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.447973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36246)
0620 14:54:03.448138 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.448209 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:54:03.462188 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27145
I20260620 14:54:03.465389 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.463114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.21.159.195:41301)
0620 14:54:03.463300 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:03.463309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:03.463331 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:54:03.463540 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:03.463552 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:03.463650 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:03.463766 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:03.464719 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:03.464725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:03.464729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:03.464747 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:03.464788 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:03.464801 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:03.464942 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:03.465020 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:54:03.465092 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:54:03.466049 27267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:03.529328 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.528765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36254)
0620 14:54:03.529016 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.529138 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:54:03.530629 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:03.530159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33495 (local address 127.0.0.1:36262)
0620 14:54:03.530377 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:03.530465 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:33495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 14:54:03.554814 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27145
I20260620 14:54:03.571323 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 26794
I20260620 14:54:03.572465 26855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:03.677459 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 26794
2026-06-20T14:54:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3007 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:54:03.720587 22143 mini_oidc.cc:152] Starting JWKS server
I20260620 14:54:03.723511 22143 webserver.cc:460] Webserver started at https://127.0.0.1:44485/ using document root <none> and password file <none>
I20260620 14:54:03.723860 22143 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:54:03.724561 22143 webserver.cc:460] Webserver started at http://127.0.0.1:44511/ using document root <none> and password file <none>
2026-06-20T14:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:03Z Disabled control of system clock
I20260620 14:54:03.753859 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:40543
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:40543
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44485/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/testchainca.cert with env {}
W20260620 14:54:04.089895 27329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:04.090210 27329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:04.090299 27329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:04.097720 27329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:04.097852 27329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:04.097891 27329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:04.097940 27329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:04.097985 27329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:54:04.098047 27329 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:54:04.098086 27329 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44485/jwks.json
I20260620 14:54:04.108278 27329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:40543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44485/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:04.109620 27329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:04.110950 27329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:04.125063 27337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:04.125160 27334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:04.125039 27335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:04.125752 27329 server_base.cc:1034] running on GCE node
I20260620 14:54:04.126684 27329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:04.128330 27329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:04.129570 27329 hybrid_clock.cc:648] HybridClock initialized: now 1781967244129510 us; error 62 us; skew 500 ppm
I20260620 14:54:04.129962 27329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:04.133249 27329 webserver.cc:460] Webserver started at http://127.21.159.254:40389/ using document root <none> and password file <none>
I20260620 14:54:04.134104 27329 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:04.134255 27329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:04.134593 27329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:04.137722 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "ed98b3b2b50a40688a6cebda7d790896"
format_stamp: "Formatted at 2026-06-20 14:54:04 on dist-test-slave-3mqx"
server_key: "e3c127b9f22c2965c075c6cb09017d2a"
server_key_iv: "5eb823c0f93635f02a3b80d35aa775da"
server_key_version: "encryptionkey@0"
I20260620 14:54:04.138589 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "ed98b3b2b50a40688a6cebda7d790896"
format_stamp: "Formatted at 2026-06-20 14:54:04 on dist-test-slave-3mqx"
server_key: "e3c127b9f22c2965c075c6cb09017d2a"
server_key_iv: "5eb823c0f93635f02a3b80d35aa775da"
server_key_version: "encryptionkey@0"
I20260620 14:54:04.144196 27329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:54:04.147886 27343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:04.149417 27329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:54:04.149638 27329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "ed98b3b2b50a40688a6cebda7d790896"
format_stamp: "Formatted at 2026-06-20 14:54:04 on dist-test-slave-3mqx"
server_key: "e3c127b9f22c2965c075c6cb09017d2a"
server_key_iv: "5eb823c0f93635f02a3b80d35aa775da"
server_key_version: "encryptionkey@0"
I20260620 14:54:04.149907 27329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:04.164248 27329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:04.165143 27329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:04.165467 27329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:04.184549 27329 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:40543
I20260620 14:54:04.184561 27394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:40543 every 8 connection(s)
I20260620 14:54:04.186028 27329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:04.190094 27395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:04.195367 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27329
I20260620 14:54:04.195598 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:04.196192 22143 external_mini_cluster.cc:1428] Setting key c9eb0d93d806034fea5fece1232b5700
I20260620 14:54:04.202744 27395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896: Bootstrap starting.
I20260620 14:54:04.206677 27395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:04.208173 27395 log.cc:824] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:04.211450 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:04.198229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.21.159.254:40543)
0620 14:54:04.199479 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:04.199495 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:04.199539 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:54:04.201314 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:04.201341 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:04.201940 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:04.202320 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:04.203101 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:04.204456 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:04.206404 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:04.207327 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:04.207484 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:04.208980 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:04.208992 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:04.209012 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:04.209606 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:04.209686 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:04.209708 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:04.209840 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:04.210102 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:54:04.210431 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":465,"threads_started":1}
I20260620 14:54:04.211498 27395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896: No bootstrap required, opened a new log
I20260620 14:54:04.215613 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:54:04.220059 27395 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed98b3b2b50a40688a6cebda7d790896" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 40543 } }
I20260620 14:54:04.220471 27395 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:04.220585 27395 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed98b3b2b50a40688a6cebda7d790896, State: Initialized, Role: FOLLOWER
I20260620 14:54:04.221223 27395 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed98b3b2b50a40688a6cebda7d790896" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 40543 } }
I20260620 14:54:04.221462 27395 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:04.221603 27395 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:04.221787 27395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:04.224089 27395 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed98b3b2b50a40688a6cebda7d790896" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 40543 } }
I20260620 14:54:04.224694 27395 leader_election.cc:302] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed98b3b2b50a40688a6cebda7d790896; no voters:
I20260620 14:54:04.225571 27395 leader_election.cc:288] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:04.225775 27400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:04.226635 27400 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [term 1 LEADER]: Becoming Leader. State: Replica: ed98b3b2b50a40688a6cebda7d790896, State: Running, Role: LEADER
I20260620 14:54:04.227201 27400 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed98b3b2b50a40688a6cebda7d790896" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 40543 } }
I20260620 14:54:04.227844 27395 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:04.230297 27401 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed98b3b2b50a40688a6cebda7d790896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed98b3b2b50a40688a6cebda7d790896" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 40543 } } }
I20260620 14:54:04.230736 27401 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:04.231251 27402 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed98b3b2b50a40688a6cebda7d790896. Latest consensus state: current_term: 1 leader_uuid: "ed98b3b2b50a40688a6cebda7d790896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed98b3b2b50a40688a6cebda7d790896" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 40543 } } }
I20260620 14:54:04.231563 27402 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:04.239990 27409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:04.245725 27409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:04.256013 27409 catalog_manager.cc:1347] Generated new cluster ID: 6c8de2e32f1443b992d2ac4812720342
I20260620 14:54:04.256160 27409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:04.268275 27409 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:04.269699 27409 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:04.282661 27409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed98b3b2b50a40688a6cebda7d790896: Generated new TSK 0
I20260620 14:54:04.283727 27409 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:54:04.325876 27425 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:54:04.326102 27425 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:54:04.327219 27398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:04.304627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.21.159.254:40543)
0620 14:54:04.304896 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:04.304902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:04.305363 (+ 461us) server_negotiation.cc:407] Connection header received
0620 14:54:04.305722 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:04.305726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:04.305795 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:04.305926 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:54:04.306594 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:04.307449 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:04.309633 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:04.310130 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:04.310227 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:04.310963 (+ 736us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:54:04.326592 (+ 15629us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:54:04.326919 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 14:54:04.327373 27398 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33606: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:54:04.329314 27424 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:04.304495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:40543 (local address 127.0.0.1:33606)
0620 14:54:04.305227 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:04.305248 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:04.305462 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:04.305996 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:04.306002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:04.306018 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:54:04.306431 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:04.306445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:04.307604 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:04.307610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:04.309472 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:04.309489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:04.310343 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:04.310350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:04.310556 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:04.310650 (+ 94us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:54:04.326945 (+ 16295us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:54:04.327397 (+ 452us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.159.254:40543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":381,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.159.254:40543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44485/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:54:04.331247 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27329
I20260620 14:54:04.332580 27390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:04.440387 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27329
2026-06-20T14:54:04Z chronyd exiting
I20260620 14:54:04.491542 22143 test_util.cc:175] -----------------------------------------------
I20260620 14:54:04.491693 22143 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967200138920-22143-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (776 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:54:04.495779 22143 mini_oidc.cc:152] Starting JWKS server
I20260620 14:54:04.497663 22143 webserver.cc:460] Webserver started at https://127.0.0.1:34873/ using document root <none> and password file <none>
I20260620 14:54:04.497922 22143 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:54:04.498602 22143 webserver.cc:460] Webserver started at http://127.0.0.1:36655/ using document root <none> and password file <none>
2026-06-20T14:54:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:04Z Disabled control of system clock
I20260620 14:54:04.529223 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:43933
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:43933
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/testchainca.cert with env {}
W20260620 14:54:04.874073 27432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:04.874375 27432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:04.874460 27432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:04.881891 27432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:04.882018 27432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:04.882056 27432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:04.882108 27432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:04.882148 27432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:54:04.882221 27432 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:54:04.882261 27432 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34873/jwks.json
I20260620 14:54:04.892614 27432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:43933
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:43933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:04.893975 27432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:04.895305 27432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:04.907855 27437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:04.907826 27440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:04.907888 27438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:04.908864 27432 server_base.cc:1034] running on GCE node
I20260620 14:54:04.909693 27432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:04.911471 27432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:04.912703 27432 hybrid_clock.cc:648] HybridClock initialized: now 1781967244912634 us; error 60 us; skew 500 ppm
I20260620 14:54:04.913054 27432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:04.915759 27432 webserver.cc:460] Webserver started at http://127.21.159.254:36117/ using document root <none> and password file <none>
I20260620 14:54:04.916488 27432 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:04.916597 27432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:04.916887 27432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:04.920186 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "427491791c18467d98e13a39ac7681d1"
format_stamp: "Formatted at 2026-06-20 14:54:04 on dist-test-slave-3mqx"
server_key: "b0829fed9679fc26229060f2b6821c75"
server_key_iv: "4b098085a1e0d18c05edc359a21846a1"
server_key_version: "encryptionkey@0"
I20260620 14:54:04.921125 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "427491791c18467d98e13a39ac7681d1"
format_stamp: "Formatted at 2026-06-20 14:54:04 on dist-test-slave-3mqx"
server_key: "b0829fed9679fc26229060f2b6821c75"
server_key_iv: "4b098085a1e0d18c05edc359a21846a1"
server_key_version: "encryptionkey@0"
I20260620 14:54:04.926937 27432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:54:04.930832 27446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:04.932540 27432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:04.932776 27432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "427491791c18467d98e13a39ac7681d1"
format_stamp: "Formatted at 2026-06-20 14:54:04 on dist-test-slave-3mqx"
server_key: "b0829fed9679fc26229060f2b6821c75"
server_key_iv: "4b098085a1e0d18c05edc359a21846a1"
server_key_version: "encryptionkey@0"
I20260620 14:54:04.932976 27432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:04.961112 27432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:04.962069 27432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:04.962405 27432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:04.982357 27432 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:43933
I20260620 14:54:04.982390 27497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:43933 every 8 connection(s)
I20260620 14:54:04.983831 27432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:04.987908 27498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:04.992313 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27432
I20260620 14:54:04.992516 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:04.993373 22143 external_mini_cluster.cc:1428] Setting key 9aa8b5c7bc53d60c08ba4ad89ca8365f
I20260620 14:54:05.000403 27498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1: Bootstrap starting.
I20260620 14:54:05.004076 27498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:05.005373 27498 log.cc:824] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:05.009035 27498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1: No bootstrap required, opened a new log
I20260620 14:54:05.009356 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:04.995632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60216 (local address 127.21.159.254:43933)
0620 14:54:04.996767 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:04.996779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:04.996825 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:54:04.998753 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:04.998787 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:04.999391 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:04.999759 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:05.000345 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.001949 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.003784 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.004660 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.004802 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:05.006341 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:05.006354 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:05.006381 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:05.007232 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:05.007317 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:05.007339 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:05.007450 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:05.007737 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 14:54:05.008145 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":419,"threads_started":1}
I20260620 14:54:05.013826 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:54:05.017784 27498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427491791c18467d98e13a39ac7681d1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43933 } }
I20260620 14:54:05.018208 27498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:05.018322 27498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 427491791c18467d98e13a39ac7681d1, State: Initialized, Role: FOLLOWER
I20260620 14:54:05.018929 27498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427491791c18467d98e13a39ac7681d1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43933 } }
I20260620 14:54:05.019245 27498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:05.019392 27498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:05.019582 27498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:05.022123 27498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427491791c18467d98e13a39ac7681d1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43933 } }
I20260620 14:54:05.022689 27498 leader_election.cc:302] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 427491791c18467d98e13a39ac7681d1; no voters:
I20260620 14:54:05.023337 27498 leader_election.cc:288] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:05.023651 27503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:05.024644 27503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [term 1 LEADER]: Becoming Leader. State: Replica: 427491791c18467d98e13a39ac7681d1, State: Running, Role: LEADER
I20260620 14:54:05.025683 27498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:05.025707 27503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427491791c18467d98e13a39ac7681d1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43933 } }
I20260620 14:54:05.029204 27505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 427491791c18467d98e13a39ac7681d1. Latest consensus state: current_term: 1 leader_uuid: "427491791c18467d98e13a39ac7681d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427491791c18467d98e13a39ac7681d1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43933 } } }
I20260620 14:54:05.029712 27505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:05.031840 27504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "427491791c18467d98e13a39ac7681d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "427491791c18467d98e13a39ac7681d1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 43933 } } }
I20260620 14:54:05.032300 27504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:05.033115 27511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:05.040701 27511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:05.050983 27511 catalog_manager.cc:1347] Generated new cluster ID: a572fbc6403848e893bca310beff014c
I20260620 14:54:05.051152 27511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:05.060958 27511 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:05.062356 27511 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:05.070822 27511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 427491791c18467d98e13a39ac7681d1: Generated new TSK 0
I20260620 14:54:05.071905 27511 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:54:05.114954 27528 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:54:05.115141 27528 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:54:05.116492 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:05.095402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.21.159.254:43933)
0620 14:54:05.095617 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:05.095625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:05.096073 (+ 448us) server_negotiation.cc:407] Connection header received
0620 14:54:05.096383 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:05.096387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:05.096464 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:05.096651 (+ 187us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:54:05.097162 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.097927 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.099482 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.100359 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.100527 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:05.101094 (+ 567us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:54:05.115750 (+ 14656us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:54:05.116167 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 14:54:05.116609 27527 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:05.095295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:43933 (local address 127.0.0.1:60230)
0620 14:54:05.095947 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:05.095964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:05.096158 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:05.096662 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:05.096668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:05.096680 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:54:05.096976 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:05.096985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.098095 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.098107 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:05.099299 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:05.099312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.100542 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.100547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:05.100698 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:05.100741 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:54:05.116038 (+ 15297us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:54:05.116265 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.159.254:43933: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":351,"threads_started":1}
W20260620 14:54:05.116645 27501 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34873/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 14:54:05.118232 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27432
I20260620 14:54:05.119727 27493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:05.216753 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27432
2026-06-20T14:54:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (777 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:54:05.273047 22143 mini_oidc.cc:152] Starting JWKS server
I20260620 14:54:05.274991 22143 webserver.cc:460] Webserver started at https://127.0.0.1:37093/ using document root <none> and password file <none>
I20260620 14:54:05.275323 22143 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:54:05.276041 22143 webserver.cc:460] Webserver started at http://127.0.0.1:44535/ using document root <none> and password file <none>
2026-06-20T14:54:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:05Z Disabled control of system clock
I20260620 14:54:05.305089 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33383
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:39177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33383
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37093/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:54:05.656486 27535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:05.656903 27535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:05.657068 27535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:05.664594 27535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:05.664791 27535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:05.664894 27535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:05.664995 27535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:05.665087 27535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:54:05.665179 27535 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:54:05.665261 27535 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37093/jwks.json
I20260620 14:54:05.674751 27535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:39177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37093/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:05.676463 27535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:05.678433 27535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:05.690526 27540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:05.690662 27541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:05.690747 27543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:05.691867 27535 server_base.cc:1034] running on GCE node
I20260620 14:54:05.692790 27535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:05.694360 27535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:05.695586 27535 hybrid_clock.cc:648] HybridClock initialized: now 1781967245695555 us; error 72 us; skew 500 ppm
I20260620 14:54:05.695999 27535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:05.698527 27535 webserver.cc:460] Webserver started at http://127.21.159.254:42419/ using document root <none> and password file <none>
I20260620 14:54:05.699461 27535 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:05.699564 27535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:05.699967 27535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:05.703400 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "045806ef43b9446996860c5af2e645f7"
format_stamp: "Formatted at 2026-06-20 14:54:05 on dist-test-slave-3mqx"
server_key: "9eb932892c98bcbfb5b650b4db99d88b"
server_key_iv: "dcb6cb42ba856470a542bf39f1ce057b"
server_key_version: "encryptionkey@0"
I20260620 14:54:05.704358 27535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "045806ef43b9446996860c5af2e645f7"
format_stamp: "Formatted at 2026-06-20 14:54:05 on dist-test-slave-3mqx"
server_key: "9eb932892c98bcbfb5b650b4db99d88b"
server_key_iv: "dcb6cb42ba856470a542bf39f1ce057b"
server_key_version: "encryptionkey@0"
I20260620 14:54:05.710260 27535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:54:05.714166 27549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:05.715698 27535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:05.715895 27535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "045806ef43b9446996860c5af2e645f7"
format_stamp: "Formatted at 2026-06-20 14:54:05 on dist-test-slave-3mqx"
server_key: "9eb932892c98bcbfb5b650b4db99d88b"
server_key_iv: "dcb6cb42ba856470a542bf39f1ce057b"
server_key_version: "encryptionkey@0"
I20260620 14:54:05.716141 27535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:05.733053 27535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:05.734042 27535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:05.734397 27535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:05.755663 27535 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:33383
I20260620 14:54:05.755681 27600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:33383 every 8 connection(s)
I20260620 14:54:05.757215 27535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:05.761418 27601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:05.767367 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27535
I20260620 14:54:05.767594 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:05.768226 22143 external_mini_cluster.cc:1428] Setting key b49318a306b296959f9c7a9ef1b3f2a1
I20260620 14:54:05.772957 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7: Bootstrap starting.
I20260620 14:54:05.776870 27601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:05.778285 27601 log.cc:824] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:05.782094 27601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7: No bootstrap required, opened a new log
I20260620 14:54:05.783881 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:05.770380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.21.159.254:33383)
0620 14:54:05.771522 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:05.771535 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:05.771582 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:54:05.773400 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:05.773426 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:05.773948 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:05.774315 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:05.775038 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.776436 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.778477 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.779747 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.779966 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:05.781279 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:05.781287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:05.781304 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:05.781890 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:05.781976 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:05.781997 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:05.782088 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:05.782401 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:54:05.782779 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":397,"threads_started":1}
I20260620 14:54:05.788123 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:54:05.791549 27601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045806ef43b9446996860c5af2e645f7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33383 } }
I20260620 14:54:05.791993 27601 raft_consensus.cc:374] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:05.792100 27601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045806ef43b9446996860c5af2e645f7, State: Initialized, Role: FOLLOWER
I20260620 14:54:05.792722 27601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045806ef43b9446996860c5af2e645f7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33383 } }
I20260620 14:54:05.792968 27601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:05.793123 27601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:05.793309 27601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:05.795656 27601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045806ef43b9446996860c5af2e645f7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33383 } }
I20260620 14:54:05.796273 27601 leader_election.cc:302] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 045806ef43b9446996860c5af2e645f7; no voters:
I20260620 14:54:05.797010 27601 leader_election.cc:288] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:05.797175 27606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:05.797935 27606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [term 1 LEADER]: Becoming Leader. State: Replica: 045806ef43b9446996860c5af2e645f7, State: Running, Role: LEADER
I20260620 14:54:05.798568 27606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045806ef43b9446996860c5af2e645f7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33383 } }
I20260620 14:54:05.799418 27601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:05.803084 27607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "045806ef43b9446996860c5af2e645f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045806ef43b9446996860c5af2e645f7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33383 } } }
I20260620 14:54:05.803562 27607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:05.803221 27608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 045806ef43b9446996860c5af2e645f7. Latest consensus state: current_term: 1 leader_uuid: "045806ef43b9446996860c5af2e645f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045806ef43b9446996860c5af2e645f7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33383 } } }
I20260620 14:54:05.803696 27608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:05.805295 27616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:05.810269 27616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:05.820537 27616 catalog_manager.cc:1347] Generated new cluster ID: 72260bd92ea24645b25367e7e65c3b70
I20260620 14:54:05.820684 27616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:05.836185 27616 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:05.837549 27616 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:05.844918 27616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 045806ef43b9446996860c5af2e645f7: Generated new TSK 0
I20260620 14:54:05.845891 27616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:05.877712 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:05.868543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.21.159.254:33383)
0620 14:54:05.868801 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:05.868810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:05.869321 (+ 511us) server_negotiation.cc:407] Connection header received
0620 14:54:05.869589 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:05.869593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:05.869676 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:05.869815 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:54:05.870455 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.871278 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.873439 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.873902 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.874000 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:05.874656 (+ 656us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:54:05.876973 (+ 2317us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:54:05.877341 (+ 368us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 14:54:05.877863 27604 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:54:05.877916 27630 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:05.868499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33383 (local address 127.0.0.1:34436)
0620 14:54:05.869183 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:05.869206 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:05.869389 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:05.869867 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:05.869872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:05.869886 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:54:05.870270 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:05.870282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.871447 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.871453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:05.873267 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:05.873285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:05.874116 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:05.874123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:05.874313 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:05.874365 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:54:05.877207 (+ 2842us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:54:05.877491 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.159.254:33383: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37093/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":368,"threads_started":1}
I20260620 14:54:05.879129 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27535
I20260620 14:54:05.880470 27596 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 14:54:05.890079 27631 webserver.cc:533] Webserver: sslize failed:
E20260620 14:54:05.890198 27631 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:54:05.972687 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27535
2026-06-20T14:54:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (755 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:06Z Disabled control of system clock
I20260620 14:54:06.056677 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33355
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33355
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/testchainca.cert with env {}
W20260620 14:54:06.401571 27636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:06.402242 27636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:06.402428 27636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:06.410830 27636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:06.411005 27636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:06.411104 27636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:06.411192 27636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:06.411279 27636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:54:06.411368 27636 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:54:06.411446 27636 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:54:06.421921 27636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:06.423465 27636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:06.424901 27636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:06.438429 27641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:06.438344 27642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:06.438333 27644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:06.439248 27636 server_base.cc:1034] running on GCE node
I20260620 14:54:06.440167 27636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:06.441744 27636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:06.442996 27636 hybrid_clock.cc:648] HybridClock initialized: now 1781967246442938 us; error 45 us; skew 500 ppm
I20260620 14:54:06.443552 27636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:06.446171 27636 webserver.cc:460] Webserver started at http://127.21.159.254:44457/ using document root <none> and password file <none>
I20260620 14:54:06.446964 27636 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:06.447129 27636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:06.447499 27636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:06.450533 27636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887"
format_stamp: "Formatted at 2026-06-20 14:54:06 on dist-test-slave-3mqx"
server_key: "ee627950504914c4adfa1094eb272387"
server_key_iv: "c37ad13f22d9abd89e43825b874a9d3e"
server_key_version: "encryptionkey@0"
I20260620 14:54:06.451458 27636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887"
format_stamp: "Formatted at 2026-06-20 14:54:06 on dist-test-slave-3mqx"
server_key: "ee627950504914c4adfa1094eb272387"
server_key_iv: "c37ad13f22d9abd89e43825b874a9d3e"
server_key_version: "encryptionkey@0"
I20260620 14:54:06.457285 27636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 14:54:06.461406 27650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:06.463124 27636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:54:06.463342 27636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887"
format_stamp: "Formatted at 2026-06-20 14:54:06 on dist-test-slave-3mqx"
server_key: "ee627950504914c4adfa1094eb272387"
server_key_iv: "c37ad13f22d9abd89e43825b874a9d3e"
server_key_version: "encryptionkey@0"
I20260620 14:54:06.463582 27636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:06.479255 27636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:06.480158 27636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:06.480455 27636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:06.500109 27636 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:33355
I20260620 14:54:06.500147 27701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:33355 every 8 connection(s)
I20260620 14:54:06.501566 27636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:06.505877 27702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:06.508985 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27636
I20260620 14:54:06.509227 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:06.510150 22143 external_mini_cluster.cc:1428] Setting key c448537a7a633eee87d03abec10d09ad
I20260620 14:54:06.519477 27702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887: Bootstrap starting.
I20260620 14:54:06.522938 27702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:06.524296 27702 log.cc:824] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:06.524564 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:06.512559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.21.159.254:33355)
0620 14:54:06.513675 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:06.513687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:06.513732 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:54:06.515503 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:06.515528 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:06.516044 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:06.516477 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:06.517135 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:06.518421 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:06.519896 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:06.520604 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:06.520820 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:06.522043 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:06.522051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:06.522066 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:06.522618 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:06.522724 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:06.522749 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:06.522898 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:06.523180 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:54:06.523542 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":392,"threads_started":1}
I20260620 14:54:06.527804 27702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887: No bootstrap required, opened a new log
I20260620 14:54:06.529112 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:54:06.537971 27702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33355 } }
I20260620 14:54:06.538435 27702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:06.538559 27702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a6ab45ec7e4a5ba8fe93a41c3bc887, State: Initialized, Role: FOLLOWER
I20260620 14:54:06.539290 27702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33355 } }
I20260620 14:54:06.539572 27702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:06.539728 27702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:06.539955 27702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:06.542459 27702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33355 } }
I20260620 14:54:06.543125 27702 leader_election.cc:302] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50a6ab45ec7e4a5ba8fe93a41c3bc887; no voters:
I20260620 14:54:06.543814 27702 leader_election.cc:288] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:06.544013 27707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:06.544711 27707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [term 1 LEADER]: Becoming Leader. State: Replica: 50a6ab45ec7e4a5ba8fe93a41c3bc887, State: Running, Role: LEADER
I20260620 14:54:06.545354 27707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33355 } }
I20260620 14:54:06.546234 27702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:06.548606 27709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50a6ab45ec7e4a5ba8fe93a41c3bc887. Latest consensus state: current_term: 1 leader_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33355 } } }
I20260620 14:54:06.548990 27709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:06.548722 27708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50a6ab45ec7e4a5ba8fe93a41c3bc887" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33355 } } }
I20260620 14:54:06.549223 27708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:06.552165 27713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:06.558080 27713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:06.568912 27713 catalog_manager.cc:1347] Generated new cluster ID: aab671fee6de4f7aa67e3ed3772d68f6
I20260620 14:54:06.569100 27713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:06.609972 27713 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:06.612023 27713 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:06.620980 27713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50a6ab45ec7e4a5ba8fe93a41c3bc887: Generated new TSK 0
I20260620 14:54:06.622212 27713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:06.699121 27705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:06.637592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.21.159.254:33355)
0620 14:54:06.637868 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:06.637876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:06.638267 (+ 391us) server_negotiation.cc:407] Connection header received
0620 14:54:06.638525 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:06.638529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:06.638609 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:06.638780 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:54:06.639380 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:06.640204 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:06.642324 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:06.642855 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:06.642970 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:06.643649 (+ 679us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:54:06.698417 (+ 54768us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:54:06.698800 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51642: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 14:54:06.699307 27705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51642: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:54:06.699903 27731 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:06.637449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33355 (local address 127.0.0.1:51642)
0620 14:54:06.638136 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:06.638158 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:06.638339 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:06.638852 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:06.638857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:06.638872 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:54:06.639214 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:06.639229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:06.640377 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:06.640383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:06.642136 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:06.642151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:06.643093 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:06.643101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:06.643301 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:06.643349 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:54:06.698739 (+ 55390us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:54:06.699111 (+ 372us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.159.254:33355: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":348,"threads_started":1}
I20260620 14:54:06.701414 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27636
I20260620 14:54:06.703361 27697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:06.804140 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27636
2026-06-20T14:54:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (811 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (428 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (427 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:07 dist-test-slave-3mqx krb5kdc[27746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:07 dist-test-slave-3mqx krb5kdc[27746](info): set up 2 sockets
Jun 20 14:54:07 dist-test-slave-3mqx krb5kdc[27746](info): commencing operation
krb5kdc: starting...
W20260620 14:54:08.859879 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:54:08 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:54:09.074208 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:41589
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:41589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:09.411360 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:09.411737 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:09.411928 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:09.419888 27759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:09.420053 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:09.420202 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:09.420322 27759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:09.420409 27759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:09.431017 27759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:41589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:41589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:09.432603 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:09.433987 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:09.446975 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:09.447129 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:09.447131 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:09.448006 27759 server_base.cc:1034] running on GCE node
I20260620 14:54:09.448886 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:09.450500 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:09.451800 27759 hybrid_clock.cc:648] HybridClock initialized: now 1781967249451741 us; error 52 us; skew 500 ppm
I20260620 14:54:09.452291 27759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:09 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967249, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:09.458953 27759 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:54:09.460675 27759 webserver.cc:460] Webserver started at http://127.21.159.254:37929/ using document root <none> and password file <none>
I20260620 14:54:09.461477 27759 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:09.461642 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:09.462005 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:09.465340 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "9066510b103142dcbd9bee5bb4ab0052"
format_stamp: "Formatted at 2026-06-20 14:54:09 on dist-test-slave-3mqx"
server_key: "95373508198bd6d284fab6c113f6514f"
server_key_iv: "5d9fe62ae88d043b86099439c970681e"
server_key_version: "encryptionkey@0"
I20260620 14:54:09.466180 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "9066510b103142dcbd9bee5bb4ab0052"
format_stamp: "Formatted at 2026-06-20 14:54:09 on dist-test-slave-3mqx"
server_key: "95373508198bd6d284fab6c113f6514f"
server_key_iv: "5d9fe62ae88d043b86099439c970681e"
server_key_version: "encryptionkey@0"
I20260620 14:54:09.471879 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 14:54:09.475960 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:09.477557 27759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:54:09.477788 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "9066510b103142dcbd9bee5bb4ab0052"
format_stamp: "Formatted at 2026-06-20 14:54:09 on dist-test-slave-3mqx"
server_key: "95373508198bd6d284fab6c113f6514f"
server_key_iv: "5d9fe62ae88d043b86099439c970681e"
server_key_version: "encryptionkey@0"
I20260620 14:54:09.478026 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:09.494954 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:09.499493 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:09.499873 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:09.519208 27759 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:41589
I20260620 14:54:09.519202 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:41589 every 8 connection(s)
I20260620 14:54:09.520750 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:09.524660 27826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:09.525836 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27759
I20260620 14:54:09.526067 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:09.526909 22143 external_mini_cluster.cc:1428] Setting key bf1d1f2233a1fcf8aed09ceb39dc7b65
I20260620 14:54:09.542471 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052: Bootstrap starting.
Jun 20 14:54:09 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:09.547159 27826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:09.548448 27826 log.cc:824] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:09.552309 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052: No bootstrap required, opened a new log
I20260620 14:54:09.556871 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:09.529180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.21.159.254:41589)
0620 14:54:09.530015 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:09.530027 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:09.530075 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:54:09.531827 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:09.531854 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:09.532404 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:09.532948 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:09.535519 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:09.537867 (+ 2348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:09.540056 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:09.541484 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:09.541705 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:09.547158 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:09.547178 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:09.547196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:09.547246 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:09.551331 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:09.552343 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:09.552354 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:09.552365 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:09.552622 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:09.553055 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:09.553061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:09.553065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:09.553897 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:09.554255 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:09.554614 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 14:54:09.555206 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":378,"threads_started":1}
I20260620 14:54:09.563166 27826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9066510b103142dcbd9bee5bb4ab0052" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41589 } }
I20260620 14:54:09.563767 27826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:09.563926 27826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9066510b103142dcbd9bee5bb4ab0052, State: Initialized, Role: FOLLOWER
I20260620 14:54:09.564687 27826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9066510b103142dcbd9bee5bb4ab0052" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41589 } }
I20260620 14:54:09.564980 27826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:09.565104 27826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:09.565243 27826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:09.567737 27826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9066510b103142dcbd9bee5bb4ab0052" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41589 } }
I20260620 14:54:09.568384 27826 leader_election.cc:302] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9066510b103142dcbd9bee5bb4ab0052; no voters:
I20260620 14:54:09.569028 27826 leader_election.cc:288] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:09.569172 27831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:09.570600 27831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [term 1 LEADER]: Becoming Leader. State: Replica: 9066510b103142dcbd9bee5bb4ab0052, State: Running, Role: LEADER
I20260620 14:54:09.571302 27831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9066510b103142dcbd9bee5bb4ab0052" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41589 } }
I20260620 14:54:09.571693 27826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:09.575366 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9066510b103142dcbd9bee5bb4ab0052. Latest consensus state: current_term: 1 leader_uuid: "9066510b103142dcbd9bee5bb4ab0052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9066510b103142dcbd9bee5bb4ab0052" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41589 } } }
I20260620 14:54:09.575832 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:09.576397 27832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9066510b103142dcbd9bee5bb4ab0052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9066510b103142dcbd9bee5bb4ab0052" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41589 } } }
I20260620 14:54:09.576800 27832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:09.578337 27840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:09.584081 27840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:09.593928 27840 catalog_manager.cc:1347] Generated new cluster ID: 776d201709f24e8391b49d7aca618578
I20260620 14:54:09.594069 27840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:09.609476 27840 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:09.610919 27840 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:09.623107 27840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9066510b103142dcbd9bee5bb4ab0052: Generated new TSK 0
I20260620 14:54:09.624289 27840 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:54:09.686838 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41589
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:10.054275 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:10.054593 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:10.054762 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:10.062513 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:10.062644 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:10.062837 27852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:10.073266 27852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:10.074774 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:10.076248 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:10.090327 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:10.090299 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:10.090406 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:10.091163 27852 server_base.cc:1034] running on GCE node
I20260620 14:54:10.091814 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:10.092798 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:10.094074 27852 hybrid_clock.cc:648] HybridClock initialized: now 1781967250094019 us; error 51 us; skew 500 ppm
I20260620 14:54:10.094476 27852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:10 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967250, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:10.098001 27852 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:54:10.099797 27852 webserver.cc:460] Webserver started at http://127.21.159.193:33755/ using document root <none> and password file <none>
I20260620 14:54:10.100574 27852 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:10.100726 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:10.101068 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:10.104255 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "b3af7e73ed694a988b6bfd2b8e8ff13e"
format_stamp: "Formatted at 2026-06-20 14:54:10 on dist-test-slave-3mqx"
server_key: "51c9a7c0693e1553bf5c71bd0be9d30f"
server_key_iv: "d04b2326e0bc2d856e9a20d2d714d522"
server_key_version: "encryptionkey@0"
I20260620 14:54:10.105404 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "b3af7e73ed694a988b6bfd2b8e8ff13e"
format_stamp: "Formatted at 2026-06-20 14:54:10 on dist-test-slave-3mqx"
server_key: "51c9a7c0693e1553bf5c71bd0be9d30f"
server_key_iv: "d04b2326e0bc2d856e9a20d2d714d522"
server_key_version: "encryptionkey@0"
I20260620 14:54:10.111238 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:54:10.115128 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:10.116720 27852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:54:10.116927 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "b3af7e73ed694a988b6bfd2b8e8ff13e"
format_stamp: "Formatted at 2026-06-20 14:54:10 on dist-test-slave-3mqx"
server_key: "51c9a7c0693e1553bf5c71bd0be9d30f"
server_key_iv: "d04b2326e0bc2d856e9a20d2d714d522"
server_key_version: "encryptionkey@0"
I20260620 14:54:10.117174 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:10.138813 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:10.144896 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:10.145238 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:10.146232 27852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:10.147686 27852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:10.147760 27852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:10.147846 27852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:10.147892 27852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:10.182691 27852 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:34101
I20260620 14:54:10.182746 27979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:34101 every 8 connection(s)
I20260620 14:54:10.184093 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:10.190243 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27852
I20260620 14:54:10.190476 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:10.191363 22143 external_mini_cluster.cc:1428] Setting key 7be38dea43143f7995765b9721c3f925
Jun 20 14:54:10 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967250, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:10.209849 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:10.186825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:51563 (local address 127.21.159.254:41589)
0620 14:54:10.187332 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:10.187337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:10.188266 (+ 929us) server_negotiation.cc:407] Connection header received
0620 14:54:10.190586 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:10.190592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:10.190676 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:10.190783 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:10.192781 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.193765 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.196534 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.197206 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.197341 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:10.202440 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:10.202450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:10.202458 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:10.202518 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:10.206171 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:10.207109 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:10.207118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:10.207124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:10.207275 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:10.207745 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:10.207753 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:10.207759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:10.208151 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:10.208309 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:10.209069 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 14:54:10.209344 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":294,"threads_started":1}
I20260620 14:54:10.210566 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:10.187348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41589 (local address 127.21.159.193:51563)
0620 14:54:10.188089 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:10.188134 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:10.190067 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:10.191140 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:10.191156 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:10.191948 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:10.192606 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:10.192625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.193933 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.193939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:10.196346 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:10.196366 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.197470 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.197480 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:10.197845 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:10.198717 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:10.198754 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:10.202105 (+ 3351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:10.206379 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:10.206388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:10.206404 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:10.206918 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:10.207387 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:10.207390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:10.207392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:10.207602 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:10.208277 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:10.208284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:10.208656 (+ 372us) client_negotiation.cc:769] Sending connection context
0620 14:54:10.208897 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 14:54:10.209334 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":296,"threads_started":1}
I20260620 14:54:10.212457 27980 heartbeater.cc:347] Connected to a master server at 127.21.159.254:41589
I20260620 14:54:10.212790 27980 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:10.213531 27980 heartbeater.cc:511] Master 127.21.159.254:41589 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:54:10.215849 27791 ts_manager.cc:194] Registered new tserver with Master: b3af7e73ed694a988b6bfd2b8e8ff13e (127.21.159.193:34101)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:54:10.217626 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:51563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:54:10.243448 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41589
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:10.582677 27986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:10.583042 27986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:10.583248 27986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:10.590977 27986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:10.591208 27986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:10.591413 27986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:10.601701 27986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:10.603263 27986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:10.604686 27986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:10.618398 27991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:10.618573 27994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:10.618582 27992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:10.619640 27986 server_base.cc:1034] running on GCE node
I20260620 14:54:10.620312 27986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:10.621474 27986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:10.622715 27986 hybrid_clock.cc:648] HybridClock initialized: now 1781967250622664 us; error 76 us; skew 500 ppm
I20260620 14:54:10.623169 27986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:10 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967250, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:10.626439 27986 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:54:10.628098 27986 webserver.cc:460] Webserver started at http://127.21.159.194:44189/ using document root <none> and password file <none>
I20260620 14:54:10.628847 27986 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:10.628996 27986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:10.629317 27986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:10.632472 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "93ddd892fb7e4757885a79baeb91addf"
format_stamp: "Formatted at 2026-06-20 14:54:10 on dist-test-slave-3mqx"
server_key: "0649f7f17bf7055216209bc2e0d44f20"
server_key_iv: "5bb6a47db06a7c718297f4370b4b46bf"
server_key_version: "encryptionkey@0"
I20260620 14:54:10.633385 27986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "93ddd892fb7e4757885a79baeb91addf"
format_stamp: "Formatted at 2026-06-20 14:54:10 on dist-test-slave-3mqx"
server_key: "0649f7f17bf7055216209bc2e0d44f20"
server_key_iv: "5bb6a47db06a7c718297f4370b4b46bf"
server_key_version: "encryptionkey@0"
I20260620 14:54:10.638809 27986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:54:10.642642 28001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:10.644315 27986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:10.644495 27986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "93ddd892fb7e4757885a79baeb91addf"
format_stamp: "Formatted at 2026-06-20 14:54:10 on dist-test-slave-3mqx"
server_key: "0649f7f17bf7055216209bc2e0d44f20"
server_key_iv: "5bb6a47db06a7c718297f4370b4b46bf"
server_key_version: "encryptionkey@0"
I20260620 14:54:10.644729 27986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:10.658607 27986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:10.663125 27986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:10.663450 27986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:10.664455 27986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:10.665880 27986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:10.665958 27986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:10.666121 27986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:10.666200 27986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:10.700783 27986 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:33297
I20260620 14:54:10.700815 28113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:33297 every 8 connection(s)
I20260620 14:54:10.702186 27986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:10.704923 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 27986
I20260620 14:54:10.705099 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:10.705657 22143 external_mini_cluster.cc:1428] Setting key 2c63dddb51dd2f783c0ab1e8cafe650a
Jun 20 14:54:10 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967250, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:10.724284 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:10.705262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:34465 (local address 127.21.159.254:41589)
0620 14:54:10.705440 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:10.705444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:10.707033 (+ 1589us) server_negotiation.cc:407] Connection header received
0620 14:54:10.709275 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:10.709284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:10.709385 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:10.709509 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:10.711135 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.712310 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.713984 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.714424 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.714568 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:10.718244 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:10.718249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:10.718253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:10.718285 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:10.721053 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:10.721869 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:10.721876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:10.721881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:10.722013 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:10.722458 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:10.722464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:10.722467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:10.722753 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:10.722915 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:10.723702 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 14:54:10.723936 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:54:10.725443 28116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:10.705823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41589 (local address 127.21.159.194:34465)
0620 14:54:10.706810 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:10.706898 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:10.708958 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:10.709747 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:10.709756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:10.710320 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:10.710919 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:10.710938 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.712474 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.712478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:10.713814 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:10.713827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:10.714681 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:10.714689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:10.714937 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:10.715738 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:10.715761 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:10.717992 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:10.721227 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:10.721239 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:10.721255 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:10.721743 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:10.722129 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:10.722134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:10.722139 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:10.722341 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:10.722891 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:10.722900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:10.723326 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 14:54:10.723601 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 14:54:10.724182 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":413,"threads_started":1}
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:54:10.727101 28114 heartbeater.cc:347] Connected to a master server at 127.21.159.254:41589
I20260620 14:54:10.727459 28114 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:10.728305 28114 heartbeater.cc:511] Master 127.21.159.254:41589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:54:10.730192 27791 ts_manager.cc:194] Registered new tserver with Master: 93ddd892fb7e4757885a79baeb91addf (127.21.159.194:33297)
I20260620 14:54:10.731133 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:34465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:54:10.752374 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41589
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:11.089625 28119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:11.089931 28119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:11.090025 28119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:11.097407 28119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:11.097528 28119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:11.097651 28119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:11.107596 28119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:11.108911 28119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:11.110155 28119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:11.123764 28125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:11.123871 28127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:11.124374 28124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:11.125361 28119 server_base.cc:1034] running on GCE node
I20260620 14:54:11.126087 28119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:11.127279 28119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:11.128520 28119 hybrid_clock.cc:648] HybridClock initialized: now 1781967251128465 us; error 52 us; skew 500 ppm
I20260620 14:54:11.128876 28119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.132480 28119 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:54:11.134023 28119 webserver.cc:460] Webserver started at http://127.21.159.195:33657/ using document root <none> and password file <none>
I20260620 14:54:11.134768 28119 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:11.134923 28119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:11.135289 28119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:11.138408 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "a9638e979381424d86ef69b58a3eab46"
format_stamp: "Formatted at 2026-06-20 14:54:11 on dist-test-slave-3mqx"
server_key: "19cb80596eee183022fde82989f33b69"
server_key_iv: "6a6c75b9b3bafd409775423f442bedca"
server_key_version: "encryptionkey@0"
I20260620 14:54:11.139323 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "a9638e979381424d86ef69b58a3eab46"
format_stamp: "Formatted at 2026-06-20 14:54:11 on dist-test-slave-3mqx"
server_key: "19cb80596eee183022fde82989f33b69"
server_key_iv: "6a6c75b9b3bafd409775423f442bedca"
server_key_version: "encryptionkey@0"
I20260620 14:54:11.144901 28119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:54:11.149854 28134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:11.151407 28119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:11.151587 28119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "a9638e979381424d86ef69b58a3eab46"
format_stamp: "Formatted at 2026-06-20 14:54:11 on dist-test-slave-3mqx"
server_key: "19cb80596eee183022fde82989f33b69"
server_key_iv: "6a6c75b9b3bafd409775423f442bedca"
server_key_version: "encryptionkey@0"
I20260620 14:54:11.151824 28119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:11.179417 28119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:11.185218 28119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:11.185539 28119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:11.186719 28119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:11.188498 28119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:11.188598 28119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:11.188721 28119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:11.188781 28119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:11.221122 27980 heartbeater.cc:503] Master 127.21.159.254:41589 was elected leader, sending a full tablet report...
I20260620 14:54:11.238631 28119 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:37209
I20260620 14:54:11.238746 28246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:37209 every 8 connection(s)
I20260620 14:54:11.240552 28119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:11.244383 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 28119
I20260620 14:54:11.244571 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:11.245250 22143 external_mini_cluster.cc:1428] Setting key 33e1aa7344c4321a08d7c203a3d91143
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:11.263759 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.243857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:46875 (local address 127.21.159.254:41589)
0620 14:54:11.244331 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.244340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.245821 (+ 1481us) server_negotiation.cc:407] Connection header received
0620 14:54:11.248567 (+ 2746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.248572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.248738 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.248844 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.250627 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.251542 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.253180 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.253654 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.253800 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.257522 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:11.257535 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:11.257538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:11.257573 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:11.260543 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.261320 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.261328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.261332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.261465 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.261910 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.261918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.261924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.262203 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:11.262353 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:11.263133 (+ 780us) server_negotiation.cc:299] Negotiation successful
0620 14:54:11.263356 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":217,"threads_started":1}
I20260620 14:54:11.264796 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.244599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41589 (local address 127.21.159.195:46875)
0620 14:54:11.245605 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:11.245653 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:11.248145 (+ 2492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:11.249136 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:11.249150 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:11.249740 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:11.250451 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:11.250470 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.251703 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.251709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:11.253016 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:11.253032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.253899 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.253905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:11.254149 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.254792 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:11.254814 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:11.257288 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:11.260689 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:11.260700 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:11.260719 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:11.261171 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:11.261579 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:11.261584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:11.261588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:11.261797 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:11.262344 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:11.262353 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:11.262727 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 14:54:11.263036 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 14:54:11.263661 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":620,"threads_started":1}
I20260620 14:54:11.266247 28247 heartbeater.cc:347] Connected to a master server at 127.21.159.254:41589
I20260620 14:54:11.266613 28247 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:11.267529 28247 heartbeater.cc:511] Master 127.21.159.254:41589 requested a full tablet report, sending...
I20260620 14:54:11.269145 27791 ts_manager.cc:194] Registered new tserver with Master: a9638e979381424d86ef69b58a3eab46 (127.21.159.195:37209)
I20260620 14:54:11.269999 27791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:46875
I20260620 14:54:11.274017 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/test-admin.keytab.
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.302573 22143 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:54:11.308004 28255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:11.305466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.21.159.193:34101)
0620 14:54:11.306100 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.306108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.306136 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:54:11.306200 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.306207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.306431 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.306582 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.307494 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41214: BlockingRecv error: recv got EOF from 127.0.0.1:41214 (error 108)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":339,"threads_started":1}
W20260620 14:54:11.308277 22143 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.309868 22143 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.311309 22143 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:54:11.327719 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.313061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.21.159.193:34101)
0620 14:54:11.313271 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.313275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.313330 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:54:11.313499 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.313502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.313563 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.313645 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.314665 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.315738 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.316972 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.318115 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.318250 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.321890 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:11.321901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:11.321917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:11.321953 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:11.324746 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.325430 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.325437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.325439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.325560 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.325901 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.325904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.325906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.326318 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:11.326644 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:11.327136 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:54:11.327447 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 14:54:11.333463 28255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:11.331810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.21.159.193:34101)
0620 14:54:11.332038 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.332043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.332184 (+ 141us) server_negotiation.cc:407] Connection header received
0620 14:54:11.332311 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.332314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.332381 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.332459 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.333210 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41224: BlockingRecv error: recv got EOF from 127.0.0.1:41224 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 14:54:11.333603 22143 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.335160 22143 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.336444 22143 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:54:11.350903 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.337791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.21.159.193:34101)
0620 14:54:11.337965 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.337970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.338326 (+ 356us) server_negotiation.cc:407] Connection header received
0620 14:54:11.338466 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.338469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.338527 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.338606 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.339548 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.340232 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.341454 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.342053 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.342166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.345815 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:11.345820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:11.345823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:11.345856 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:11.348532 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.349251 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.349255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.349258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.349381 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.349702 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.349705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.349707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.349967 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:11.350158 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:11.350381 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:54:11.350551 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 14:54:11.355302 28255 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:11.353492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.21.159.193:34101)
0620 14:54:11.353707 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.353712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.353977 (+ 265us) server_negotiation.cc:407] Connection header received
0620 14:54:11.354171 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.354176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.354254 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.354355 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.355038 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41242: BlockingRecv error: recv got EOF from 127.0.0.1:41242 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 14:54:11.355546 22143 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967200138920-22143-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.357115 22143 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:11.358446 22143 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:54:11.373023 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.359918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.21.159.193:34101)
0620 14:54:11.360145 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.360152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.360422 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:54:11.360631 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.360635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.360694 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.360797 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.361770 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.362521 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.363836 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.364270 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.364424 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.368126 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:11.368131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:11.368134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:11.368170 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:11.370542 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.371211 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.371215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.371218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.371352 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.371732 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.371735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.371737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.372035 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:11.372270 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:11.372533 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 14:54:11.372697 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:54:11.374763 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27852
I20260620 14:54:11.387727 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.375707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.21.159.193:34101)
0620 14:54:11.375921 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.375926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.376143 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:54:11.376350 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.376360 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.376420 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.376548 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.377663 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.378349 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.379728 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.380161 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.380301 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.382481 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:11.382485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:11.382488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:11.382521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:11.385131 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.385778 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.385790 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.385793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.385941 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.386323 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.386330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.386336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.386593 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:11.386881 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:11.387200 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:54:11.387414 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:54:11.388427 27975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:11.473935 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27852
I20260620 14:54:11.490571 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27986
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:54:11.506131 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.491585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.21.159.194:33297)
0620 14:54:11.492154 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.492161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.492185 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:54:11.492245 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.492251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.492497 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.492622 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.493600 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.494688 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.496141 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.496731 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.496829 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.500711 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:11.500723 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:11.500738 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:11.500772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:11.503605 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.504273 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.504280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.504283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.504406 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.504708 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.504711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.504713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.505040 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:11.505263 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:11.505519 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:54:11.505784 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":292,"threads_started":1}
I20260620 14:54:11.507603 28109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:11.591917 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27986
I20260620 14:54:11.608129 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 28119
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:54:11.624536 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:11.609192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.21.159.195:37209)
0620 14:54:11.609918 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:11.609925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:11.609960 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:54:11.610022 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:11.610028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:11.610245 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:11.610373 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:11.611515 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.612548 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.613778 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:11.614393 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:11.614503 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:11.618536 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:11.618549 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:11.618565 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:11.618604 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:11.621568 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.622242 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.622249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.622251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.622371 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:11.622711 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:11.622715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:11.622717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:11.623143 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:11.623372 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:11.623849 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 14:54:11.624207 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":419,"threads_started":1}
I20260620 14:54:11.625924 28242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:11.710222 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 28119
I20260620 14:54:11.725986 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 27759
I20260620 14:54:11.727491 27821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:11.830013 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 27759
2026-06-20T14:54:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4176 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[28275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[28275](info): set up 2 sockets
Jun 20 14:54:11 dist-test-slave-3mqx krb5kdc[28275](info): commencing operation
krb5kdc: starting...
W20260620 14:54:12.963936 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:54:13 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab.
Entry for principal oryx/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab.
I20260620 14:54:13.191112 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:32891
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab
--principal=oryx/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:13.527963 28288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:13.528352 28288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:13.528528 28288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:13.535923 28288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:13.536124 28288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:13.536218 28288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:13.536331 28288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:13.536417 28288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:13.546515 28288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab
--principal=oryx/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:13.547984 28288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:13.549369 28288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:13.562829 28293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:13.562908 28296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:13.562865 28294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:13.563758 28288 server_base.cc:1034] running on GCE node
I20260620 14:54:13.564604 28288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:13.566299 28288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:13.567550 28288 hybrid_clock.cc:648] HybridClock initialized: now 1781967253567515 us; error 77 us; skew 500 ppm
I20260620 14:54:13.567932 28288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:13 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:13.571530 28288 init.cc:377] Logged in from keytab as oryx/127.21.159.254@KRBTEST.COM (short username oryx)
I20260620 14:54:13.573232 28288 webserver.cc:460] Webserver started at http://127.21.159.254:44473/ using document root <none> and password file <none>
I20260620 14:54:13.573983 28288 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:13.574146 28288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:13.574473 28288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:13.577755 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "5c179d685f7f4ba68e8a0760536cbb73"
format_stamp: "Formatted at 2026-06-20 14:54:13 on dist-test-slave-3mqx"
server_key: "5502fe438f7ce4be328f3c095bcfdbc2"
server_key_iv: "888b34e03c627337045ad2a83f8d6333"
server_key_version: "encryptionkey@0"
I20260620 14:54:13.578616 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "5c179d685f7f4ba68e8a0760536cbb73"
format_stamp: "Formatted at 2026-06-20 14:54:13 on dist-test-slave-3mqx"
server_key: "5502fe438f7ce4be328f3c095bcfdbc2"
server_key_iv: "888b34e03c627337045ad2a83f8d6333"
server_key_version: "encryptionkey@0"
I20260620 14:54:13.584347 28288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:54:13.588073 28303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:13.589610 28288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:54:13.589838 28288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "5c179d685f7f4ba68e8a0760536cbb73"
format_stamp: "Formatted at 2026-06-20 14:54:13 on dist-test-slave-3mqx"
server_key: "5502fe438f7ce4be328f3c095bcfdbc2"
server_key_iv: "888b34e03c627337045ad2a83f8d6333"
server_key_version: "encryptionkey@0"
I20260620 14:54:13.590082 28288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:13.625125 28288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:13.630066 28288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:13.630388 28288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:13.649691 28288 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:32891
I20260620 14:54:13.649721 28354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:32891 every 8 connection(s)
I20260620 14:54:13.651328 28288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:13.652853 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 28288
I20260620 14:54:13.653075 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:13.653899 22143 external_mini_cluster.cc:1428] Setting key 7f28d469a556ce9418a5162371e5f1e8
I20260620 14:54:13.656550 28355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:54:13 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:13.671202 28355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: Bootstrap starting.
I20260620 14:54:13.675673 28355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:13.677031 28355 log.cc:824] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:13.679666 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:13.656120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.21.159.254:32891)
0620 14:54:13.656997 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:13.657010 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:13.657053 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:54:13.658648 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:13.658669 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:13.659177 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:13.659552 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:13.661464 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:13.663042 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:13.664943 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:13.665670 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:13.665805 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:13.670629 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:13.670646 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:13.670668 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:13.670720 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:13.674754 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:13.675589 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:13.675599 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:13.675608 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:13.675841 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:13.676235 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:13.676240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:13.676243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:13.676972 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:13.677300 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:13.677810 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 14:54:13.678287 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":409,"threads_started":1}
I20260620 14:54:13.680651 28355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: No bootstrap required, opened a new log
I20260620 14:54:13.688611 28355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } }
I20260620 14:54:13.689008 28355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:13.689086 28355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c179d685f7f4ba68e8a0760536cbb73, State: Initialized, Role: FOLLOWER
I20260620 14:54:13.689678 28355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } }
I20260620 14:54:13.689899 28355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:13.689999 28355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:13.690135 28355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:13.692898 28355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } }
I20260620 14:54:13.693497 28355 leader_election.cc:302] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c179d685f7f4ba68e8a0760536cbb73; no voters:
I20260620 14:54:13.694175 28355 leader_election.cc:288] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:13.694288 28360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:13.695047 28360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 1 LEADER]: Becoming Leader. State: Replica: 5c179d685f7f4ba68e8a0760536cbb73, State: Running, Role: LEADER
I20260620 14:54:13.695698 28360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } }
I20260620 14:54:13.696467 28355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:13.698441 28361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } }
I20260620 14:54:13.698822 28361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:13.699280 28362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c179d685f7f4ba68e8a0760536cbb73. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } }
I20260620 14:54:13.699597 28362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:13.701560 28368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:13.707674 28368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:13.717216 28368 catalog_manager.cc:1347] Generated new cluster ID: 4ef396ced08041b484a5914dc262644d
I20260620 14:54:13.717391 28368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:13.736683 28368 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:13.738235 28368 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:13.750347 28368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: Generated new TSK 0
I20260620 14:54:13.751456 28368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab.
Entry for principal oryx/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab.
I20260620 14:54:13.819092 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32891
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab
--principal=oryx/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:14.179580 28381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:14.179926 28381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:14.180104 28381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:14.187625 28381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:14.187795 28381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:14.187966 28381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:14.198408 28381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab
--principal=oryx/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:14.199899 28381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:14.201246 28381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:14.215778 28386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:14.215751 28389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:14.216107 28387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:14.217070 28381 server_base.cc:1034] running on GCE node
I20260620 14:54:14.217860 28381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:14.218951 28381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:14.220201 28381 hybrid_clock.cc:648] HybridClock initialized: now 1781967254220120 us; error 71 us; skew 500 ppm
I20260620 14:54:14.220543 28381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:14 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967254, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:14.224051 28381 init.cc:377] Logged in from keytab as oryx/127.21.159.193@KRBTEST.COM (short username oryx)
I20260620 14:54:14.225850 28381 webserver.cc:460] Webserver started at http://127.21.159.193:34189/ using document root <none> and password file <none>
I20260620 14:54:14.226675 28381 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:14.226843 28381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:14.227259 28381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:14.230675 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "0613f1ae65d84469b92626f98ee51cc2"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "0c1b5038484a3ac8fbc87a543098f9e0"
server_key_iv: "ef66d9ce1e426d933142915b48d44e32"
server_key_version: "encryptionkey@0"
I20260620 14:54:14.231680 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "0613f1ae65d84469b92626f98ee51cc2"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "0c1b5038484a3ac8fbc87a543098f9e0"
server_key_iv: "ef66d9ce1e426d933142915b48d44e32"
server_key_version: "encryptionkey@0"
I20260620 14:54:14.238042 28381 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 14:54:14.242058 28396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:14.243678 28381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:14.243906 28381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "0613f1ae65d84469b92626f98ee51cc2"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "0c1b5038484a3ac8fbc87a543098f9e0"
server_key_iv: "ef66d9ce1e426d933142915b48d44e32"
server_key_version: "encryptionkey@0"
I20260620 14:54:14.244124 28381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:14.257879 28381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:14.262697 28381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:14.263023 28381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:14.264072 28381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:14.265441 28381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:14.265515 28381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:14.265689 28381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:14.265753 28381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:14.299113 28381 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:35161
I20260620 14:54:14.299229 28508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:35161 every 8 connection(s)
I20260620 14:54:14.300482 28381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:14.303941 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 28381
I20260620 14:54:14.304174 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:14.305014 22143 external_mini_cluster.cc:1428] Setting key 26317a12626010e2d1e2507e1ab2d3ca
Jun 20 14:54:14 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967254, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:14.325106 28511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:14.303199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:59839 (local address 127.21.159.254:32891)
0620 14:54:14.303663 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:14.303667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:14.304764 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 14:54:14.306829 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:14.306833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:14.306914 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:14.307047 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:14.308696 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.309883 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.312688 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.313093 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.313200 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:14.317772 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:14.317778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:14.317781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:14.317819 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:14.320648 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:14.321825 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:14.321831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:14.321833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:14.321977 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:14.322631 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:14.322636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:14.322638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:14.322957 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:14.323166 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:14.324324 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0620 14:54:14.324615 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20260620 14:54:14.326696 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:14.303792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.193:59839)
0620 14:54:14.304579 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:14.304618 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:14.306490 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:14.307356 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:14.307365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:14.307909 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:14.308513 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:14.308532 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.310124 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.310133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:14.312472 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:14.312495 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.313371 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.313386 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:14.313602 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:14.314435 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:14.314465 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:14.317431 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:14.320835 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:14.320851 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:14.320875 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:14.321643 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:14.322124 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:14.322133 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:14.322139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:14.322449 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:14.323193 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:14.323205 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:14.323794 (+ 589us) client_negotiation.cc:769] Sending connection context
0620 14:54:14.324169 (+ 375us) client_negotiation.cc:240] Negotiation successful
0620 14:54:14.325043 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for oryx/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:54:14.328665 28509 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32891
I20260620 14:54:14.329041 28509 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:14.329873 28509 heartbeater.cc:511] Master 127.21.159.254:32891 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.194@KRBTEST.COM" created.
I20260620 14:54:14.332430 28320 ts_manager.cc:194] Registered new tserver with Master: 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161)
I20260620 14:54:14.334035 28320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:59839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab.
Entry for principal oryx/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab.
I20260620 14:54:14.358484 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32891
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab
--principal=oryx/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:14.722246 28515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:14.722632 28515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:14.722801 28515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:14.730741 28515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:14.730983 28515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:14.731215 28515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:14.743101 28515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab
--principal=oryx/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:14.744671 28515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:14.746218 28515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:14.761859 28520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:14.761957 28523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:14.761924 28521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:14.763334 28515 server_base.cc:1034] running on GCE node
I20260620 14:54:14.764108 28515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:14.765251 28515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:14.766525 28515 hybrid_clock.cc:648] HybridClock initialized: now 1781967254766495 us; error 55 us; skew 500 ppm
I20260620 14:54:14.767007 28515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:14 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967254, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:14.774142 28515 init.cc:377] Logged in from keytab as oryx/127.21.159.194@KRBTEST.COM (short username oryx)
I20260620 14:54:14.775928 28515 webserver.cc:460] Webserver started at http://127.21.159.194:40481/ using document root <none> and password file <none>
I20260620 14:54:14.776697 28515 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:14.776846 28515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:14.777172 28515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:14.780318 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "bb466bdeae9b45daa55de809056af60e"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "609f74381a55a39acca214169059e7db"
server_key_iv: "f0edb850b9b03d8a715e5afe4d43c743"
server_key_version: "encryptionkey@0"
I20260620 14:54:14.781128 28515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "bb466bdeae9b45daa55de809056af60e"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "609f74381a55a39acca214169059e7db"
server_key_iv: "f0edb850b9b03d8a715e5afe4d43c743"
server_key_version: "encryptionkey@0"
I20260620 14:54:14.786669 28515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:54:14.790776 28530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:14.792407 28515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:14.792609 28515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "bb466bdeae9b45daa55de809056af60e"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "609f74381a55a39acca214169059e7db"
server_key_iv: "f0edb850b9b03d8a715e5afe4d43c743"
server_key_version: "encryptionkey@0"
I20260620 14:54:14.792876 28515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:14.816133 28515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:14.821592 28515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:14.821992 28515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:14.823194 28515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:14.824785 28515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:14.824908 28515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:14.825078 28515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:14.825162 28515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:14.863371 28515 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:41545
I20260620 14:54:14.863418 28642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:41545 every 8 connection(s)
I20260620 14:54:14.864828 28515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:14.872329 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 28515
I20260620 14:54:14.872576 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:14.873064 22143 external_mini_cluster.cc:1428] Setting key 4ab55e12307f89b0e6883e3cba73cdf1
Jun 20 14:54:14 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967254, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:14.887670 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:14.867794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:35957 (local address 127.21.159.254:32891)
0620 14:54:14.868309 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:14.868315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:14.869340 (+ 1025us) server_negotiation.cc:407] Connection header received
0620 14:54:14.871328 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:14.871334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:14.871448 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:14.871584 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:14.873315 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.874335 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.876402 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.876882 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.877041 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:14.881029 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:14.881034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:14.881038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:14.881072 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:14.884019 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:14.884893 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:14.884899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:14.884901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:14.885072 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:14.885687 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:14.885692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:14.885695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:14.885976 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:14.886096 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:14.887172 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0620 14:54:14.887365 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
I20260620 14:54:14.889338 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:14.868386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:35957)
0620 14:54:14.869155 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:14.869203 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:14.870975 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:14.871846 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:14.871855 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:14.872379 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:14.873079 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:14.873098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.874525 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.874530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:14.876226 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:14.876241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:14.877153 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:14.877164 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:14.877441 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:14.878234 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:14.878259 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:14.880759 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:14.884206 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:14.884216 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:14.884237 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:14.884739 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:14.885238 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:14.885246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:14.885252 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:14.885541 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:14.886126 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:14.886138 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:14.886661 (+ 523us) client_negotiation.cc:769] Sending connection context
0620 14:54:14.886985 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 14:54:14.887683 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":323,"threads_started":1}
I20260620 14:54:14.891206 28643 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32891
I20260620 14:54:14.891569 28643 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:14.892470 28643 heartbeater.cc:511] Master 127.21.159.254:32891 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:54:14.894354 28320 ts_manager.cc:194] Registered new tserver with Master: bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545)
I20260620 14:54:14.895462 28320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:35957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab.
Entry for principal oryx/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab.
I20260620 14:54:14.924636 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32891
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab
--principal=oryx/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:15.269160 28649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:15.269451 28649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:15.269629 28649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:15.276934 28649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:15.277050 28649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:15.277258 28649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:15.287175 28649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab
--principal=oryx/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:15.288574 28649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:15.289975 28649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:15.304483 28657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:15.304560 28654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:15.304512 28655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:15.305752 28649 server_base.cc:1034] running on GCE node
I20260620 14:54:15.306437 28649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:15.307451 28649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:15.308698 28649 hybrid_clock.cc:648] HybridClock initialized: now 1781967255308659 us; error 55 us; skew 500 ppm
I20260620 14:54:15.309044 28649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:15 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:15.312351 28649 init.cc:377] Logged in from keytab as oryx/127.21.159.195@KRBTEST.COM (short username oryx)
I20260620 14:54:15.313915 28649 webserver.cc:460] Webserver started at http://127.21.159.195:39029/ using document root <none> and password file <none>
I20260620 14:54:15.314646 28649 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:15.314797 28649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:15.315171 28649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:15.318269 28649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "687f2d289d8d44a884e8c417758b1413"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "d05a180e5b95064ade60543a4d85d938"
server_key_iv: "216da50db637446cd59653aa7ea94cc5"
server_key_version: "encryptionkey@0"
I20260620 14:54:15.319211 28649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "687f2d289d8d44a884e8c417758b1413"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "d05a180e5b95064ade60543a4d85d938"
server_key_iv: "216da50db637446cd59653aa7ea94cc5"
server_key_version: "encryptionkey@0"
I20260620 14:54:15.324856 28649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 14:54:15.328866 28664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:15.330581 28649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:54:15.330822 28649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "687f2d289d8d44a884e8c417758b1413"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "d05a180e5b95064ade60543a4d85d938"
server_key_iv: "216da50db637446cd59653aa7ea94cc5"
server_key_version: "encryptionkey@0"
I20260620 14:54:15.331153 28649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:15.337867 28509 heartbeater.cc:503] Master 127.21.159.254:32891 was elected leader, sending a full tablet report...
I20260620 14:54:15.385526 28649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:15.390302 28649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:15.390645 28649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:15.391815 28649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:15.393321 28649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:15.393445 28649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:15.393622 28649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:15.393704 28649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:15.430445 28649 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:43335
I20260620 14:54:15.430488 28776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:43335 every 8 connection(s)
I20260620 14:54:15.432037 28649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:15.438761 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 28649
I20260620 14:54:15.438962 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:15.439472 22143 external_mini_cluster.cc:1428] Setting key fa70322471bf2c60f44a7e1067aff312
Jun 20 14:54:15 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:15.453843 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:15.434744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:53921 (local address 127.21.159.254:32891)
0620 14:54:15.435223 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:15.435228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:15.436394 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 14:54:15.438264 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:15.438269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:15.438396 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:15.438521 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:15.440364 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.441351 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.443178 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.443569 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.443665 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:15.447398 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:15.447403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:15.447407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:15.447441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:15.450377 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:15.451371 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:15.451375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:15.451378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:15.451506 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:15.451932 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:15.451935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:15.451937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:15.452208 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:15.452318 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:15.453199 (+ 881us) server_negotiation.cc:299] Negotiation successful
0620 14:54:15.453425 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":227,"threads_started":1}
I20260620 14:54:15.455235 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:15.435376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.195:53921)
0620 14:54:15.436217 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:15.436257 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:15.437970 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:15.438786 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:15.438795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:15.439453 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:15.440158 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:15.440185 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.441516 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.441536 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:15.442993 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:15.443007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.443759 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.443764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:15.443982 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:15.444712 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:15.444735 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:15.447130 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:15.450564 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:15.450572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:15.450588 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:15.451225 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:15.451621 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:15.451625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:15.451628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:15.451828 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:15.452345 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:15.452352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:15.452727 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 14:54:15.453073 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 14:54:15.453691 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":373,"threads_started":1}
I20260620 14:54:15.456918 28777 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32891
I20260620 14:54:15.457247 28777 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:15.457976 28777 heartbeater.cc:511] Master 127.21.159.254:32891 requested a full tablet report, sending...
I20260620 14:54:15.459493 28320 ts_manager.cc:194] Registered new tserver with Master: 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335)
I20260620 14:54:15.460400 28320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:53921
I20260620 14:54:15.468183 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.159.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.253.keytab.
Entry for principal oryx/127.21.159.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.253.keytab.
I20260620 14:54:15.518936 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:35607
--webserver_interface=127.21.159.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.253.keytab
--principal=oryx/127.21.159.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:32891,127.21.159.253:35607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:15.865604 28783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:15.865896 28783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:15.866071 28783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:15.873781 28783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:15.873912 28783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:15.874042 28783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:15.874132 28783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:15.874214 28783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:15.884696 28783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:32891,127.21.159.253:35607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.253.keytab
--principal=oryx/127.21.159.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:35607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.159.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
W20260620 14:54:15.885133 28783 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.159.254:32891,127.21.159.253:35607'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:54:15.886248 28783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:15.887708 28783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:54:15.898913 28643 heartbeater.cc:503] Master 127.21.159.254:32891 was elected leader, sending a full tablet report...
W20260620 14:54:15.901638 28788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:15.901597 28791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:15.901542 28789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:15.902787 28783 server_base.cc:1034] running on GCE node
I20260620 14:54:15.903712 28783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:15.905233 28783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:15.906489 28783 hybrid_clock.cc:648] HybridClock initialized: now 1781967255906410 us; error 57 us; skew 500 ppm
I20260620 14:54:15.906896 28783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:15 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:15.910233 28783 init.cc:377] Logged in from keytab as oryx/127.21.159.253@KRBTEST.COM (short username oryx)
I20260620 14:54:15.911998 28783 webserver.cc:460] Webserver started at http://127.21.159.253:40671/ using document root <none> and password file <none>
I20260620 14:54:15.912767 28783 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:15.912902 28783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:15.913232 28783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:15.916390 28783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/instance:
uuid: "dc4a2c20c0a4469da2830f9ad5355817"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "87fb4050ded8c9692d3d6388f282c695"
server_key_iv: "9a82f1a784ce687c0700b225e6dcb5b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:15.917199 28783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal/instance:
uuid: "dc4a2c20c0a4469da2830f9ad5355817"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "87fb4050ded8c9692d3d6388f282c695"
server_key_iv: "9a82f1a784ce687c0700b225e6dcb5b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:15.922525 28783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:54:15.926139 28798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:15.927686 28783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:54:15.927886 28783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
uuid: "dc4a2c20c0a4469da2830f9ad5355817"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "87fb4050ded8c9692d3d6388f282c695"
server_key_iv: "9a82f1a784ce687c0700b225e6dcb5b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:15.928123 28783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:15.940936 28783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:15.945113 28783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:15.945407 28783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:15.963918 28783 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.253:35607
I20260620 14:54:15.963938 28849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.253:35607 every 8 connection(s)
I20260620 14:54:15.965411 28783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
I20260620 14:54:15.969475 28850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:15.972054 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 28783
I20260620 14:54:15.977375 28850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:54:15 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.253@KRBTEST.COM
Jun 20 14:54:15 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.253@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:15.997145 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:15.980434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.21.159.254:32891)
0620 14:54:15.981811 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:15.981816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:15.981832 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:54:15.981882 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:15.981886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:15.981944 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:15.982088 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:15.983395 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.984299 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.986006 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.986412 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.986566 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:15.990714 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:15.990721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:15.990726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:15.990776 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:15.993856 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:15.994755 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:15.994761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:15.994764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:15.994895 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:15.995368 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:15.995372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:15.995376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:15.995788 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:15.995941 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:15.996607 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 14:54:15.996842 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":1134,"threads_started":1}
I20260620 14:54:15.998692 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:15.974608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.21.159.253:35607)
0620 14:54:15.975535 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:15.975548 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:15.975593 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:54:15.977475 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:15.977501 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:15.977989 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:15.978422 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:15.980270 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.982281 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.983676 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.984372 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.984600 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:15.989233 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:15.989246 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:15.989268 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:15.989319 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:15.993178 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:15.993977 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:15.993988 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:15.993997 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:15.994212 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:15.994635 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:15.994638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:15.994640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:15.995477 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:15.995804 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:15.996697 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 14:54:15.997132 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":703,"thread_start_us":456,"threads_started":1}
I20260620 14:54:15.998687 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:15.980343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:37624)
0620 14:54:15.981171 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:15.981222 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:15.981374 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:15.982156 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:15.982164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:15.982876 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:15.983227 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:15.983238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.984464 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.984472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:15.985844 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:15.985857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:15.986666 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:15.986675 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:15.986838 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:15.987615 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:15.987658 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:15.990488 (+ 2830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:15.994001 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:15.994010 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:15.994016 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:15.994630 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:15.995025 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:15.995029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:15.995031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:15.995264 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:15.995944 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:15.995952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:15.996178 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 14:54:15.996483 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 14:54:15.997228 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":153,"thread_start_us":368,"threads_started":1}
I20260620 14:54:16.001645 28850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967255, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.253@KRBTEST.COM for oryx/127.21.159.253@KRBTEST.COM
I20260620 14:54:16.015260 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.002601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.0.0.1:53230)
0620 14:54:16.002767 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.002782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.002907 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.003260 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.003264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.003748 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.003925 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.003934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.004921 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.004926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.005792 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.005802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.006491 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.006496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.006737 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.007353 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.007371 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.009558 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.012796 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.012800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.012803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.013199 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.013600 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.013603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.013605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.013755 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.014452 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.014455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.014574 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.014687 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.014927 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":1}
I20260620 14:54:16.015493 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.002478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.21.159.253:35607)
0620 14:54:16.002653 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.002660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.002848 (+ 188us) server_negotiation.cc:407] Connection header received
0620 14:54:16.003028 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.003033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.003121 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.003246 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.004076 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.004780 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.005929 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.006349 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.006483 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.009778 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.009783 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.009787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.009817 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.012642 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.013327 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.013331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.013334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.013481 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.013893 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.013898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.013900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.014334 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.014472 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.014807 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.015204 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":53}
I20260620 14:54:16.020396 28850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Bootstrap starting.
I20260620 14:54:16.023789 28850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:16.024906 28850 log.cc:824] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:16.027951 28850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: No bootstrap required, opened a new log
I20260620 14:54:16.035755 28850 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } }
I20260620 14:54:16.036119 28850 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:16.036252 28850 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc4a2c20c0a4469da2830f9ad5355817, State: Initialized, Role: FOLLOWER
I20260620 14:54:16.036972 28850 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } }
I20260620 14:54:16.037875 28857 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } } }
I20260620 14:54:16.038229 28857 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:54:16.038978 28850 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:54:16.048542 28868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:54:16.048736 28868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:54:16.051647 28783 master_runner.cc:428] Detected that this master dc4a2c20c0a4469da2830f9ad5355817 is joining an existing cluster
I20260620 14:54:16.051793 28783 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.159.253:35607
WARNING: no policy specified for oryx/127.21.159.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.252@KRBTEST.COM" created.
I20260620 14:54:16.083237 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.056735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:37638)
0620 14:54:16.063753 (+ 7018us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.063792 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.064137 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.064876 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.064884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.065860 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.066419 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.066437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.068539 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.068548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.070936 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.070961 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.072830 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.072839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.073129 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.074125 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.074162 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.075671 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.079987 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.079994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.079999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.080524 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.080961 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.080966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.080970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.081161 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.081883 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.081891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.082122 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.082266 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.082684 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6759,"thread_start_us":6516,"threads_started":1}
I20260620 14:54:16.083745 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.057234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.21.159.254:32891)
0620 14:54:16.057681 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.057690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.063958 (+ 6268us) server_negotiation.cc:407] Connection header received
0620 14:54:16.064330 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.064339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.064469 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.064559 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.066716 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.068266 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.071223 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.072223 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.072379 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.075946 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.075954 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.075958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.076001 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.079817 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.080679 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.080685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.080689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.080834 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.081320 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.081325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.081328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.081735 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.082775 (+ 1040us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.082928 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.083217 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:54:16.092641 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.056691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.21.159.253:35607)
0620 14:54:16.056881 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.056887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.075896 (+ 19009us) server_negotiation.cc:407] Connection header received
0620 14:54:16.076072 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.076079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.076164 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.076249 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.077674 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.078843 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.080568 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.081352 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.081480 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.085352 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.085359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.085364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.085412 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.089485 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.090325 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.090331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.090334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.090468 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.090885 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.090891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.090898 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.091350 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.091489 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.092023 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.092278 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:54:16.093906 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.057207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.0.0.1:53236)
0620 14:54:16.075668 (+ 18461us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.075737 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.075909 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.076455 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.076461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.077108 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.077479 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.077504 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.079177 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.079184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.080416 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.080428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.081563 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.081570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.081772 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.082460 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.082482 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.085132 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.089660 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.089665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.089670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.090189 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.090611 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.090615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.090619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.090774 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.091562 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.091569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.091721 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.093043 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.093406 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18252,"thread_start_us":17991,"threads_started":1}
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.254@KRBTEST.COM for oryx/127.21.159.253@KRBTEST.COM
I20260620 14:54:16.113164 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.091467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.21.159.253:35607)
0620 14:54:16.093007 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.093012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.093956 (+ 944us) server_negotiation.cc:407] Connection header received
0620 14:54:16.094385 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.094394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.094566 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.094659 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.096547 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.097744 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.099818 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.100605 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.100737 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.105565 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.105573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.105578 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.105625 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.109624 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.110691 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.110697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.110701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.110862 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.111364 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.111369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.111373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.111716 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.111861 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.112602 (+ 741us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.112857 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1403,"thread_start_us":13292,"threads_started":1}
I20260620 14:54:16.113278 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.091044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.0.0.1:53248)
0620 14:54:16.093511 (+ 2467us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.093553 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.093759 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.094868 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.094878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.095938 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.096354 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.096371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.097916 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.097922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.099634 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.099651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.100985 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.100992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.101228 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.102133 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.102160 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.105321 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.109827 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.109845 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.109854 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.110520 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.111009 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.111014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.111018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.111224 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.111944 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.111956 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.112232 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.112459 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.112834 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2164,"thread_start_us":2018,"threads_started":1}
I20260620 14:54:16.115526 28320 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.159.253:35607: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817"
member_type: NON_VOTER
last_known_addr {
host: "127.21.159.253"
port: 35607
}
attrs {
promote: true
}
}
dest_uuid: "5c179d685f7f4ba68e8a0760536cbb73"
cas_config_opid_index: -1
I20260620 14:54:16.116699 28320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } }
I20260620 14:54:16.120193 28882 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dc4a2c20c0a4469da2830f9ad5355817 (127.21.159.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } }
I20260620 14:54:16.121349 28825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:16.124701 28882 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.159.253:35607
I20260620 14:54:16.125245 28825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Refusing update from remote peer 5c179d685f7f4ba68e8a0760536cbb73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.252.keytab.
Entry for principal oryx/127.21.159.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.252.keytab.
I20260620 14:54:16.125578 28783 master.cc:537] Master@127.21.159.253:35607 shutting down...
I20260620 14:54:16.126722 28883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:16.127285 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:39419
--webserver_interface=127.21.159.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.252.keytab
--principal=oryx/127.21.159.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:16.130369 28307 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 14:54:16.130805 28881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } }
I20260620 14:54:16.131249 28881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
W20260620 14:54:16.142792 28307 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 -> Peer dc4a2c20c0a4469da2830f9ad5355817 (127.21.159.253:35607): Couldn't send request to peer dc4a2c20c0a4469da2830f9ad5355817. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:54:16.145936 28783 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:16.146442 28783 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:16.146689 28783 tablet_replica.cc:330] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: stopping tablet replica
I20260620 14:54:16.154325 28783 master.cc:552] Master@127.21.159.253:35607 shutdown complete.
I20260620 14:54:16.154522 28783 master_runner.cc:305] Clearing existing system tablet
I20260620 14:54:16.155114 28783 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:16.155329 28783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:16.159471 28783 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:54:16.161650 28888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:16.162050 28783 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:16.162335 28783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:54:16.162431 28783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
uuid: "dc4a2c20c0a4469da2830f9ad5355817"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "87fb4050ded8c9692d3d6388f282c695"
server_key_iv: "9a82f1a784ce687c0700b225e6dcb5b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:16.168310 28783 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:16.176374 28783 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:54:16.176651 28783 log.cc:1196] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:54:16.177197 28783 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Deleting consensus metadata
I20260620 14:54:16.177523 28783 master_runner.cc:315] Copying system tablet from 127.21.159.254:32891
I20260620 14:54:16.180330 28783 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: tablet copy: Beginning tablet copy session from remote peer at address 127.21.159.254:32891
I20260620 14:54:16.199883 28894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.181281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:37654)
0620 14:54:16.183559 (+ 2278us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.183584 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.183774 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.184234 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.184240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.184805 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.185115 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.185130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.186660 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.186666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.188372 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.188389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.189487 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.189494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.189721 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.190522 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.190549 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.191861 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.196315 (+ 4454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.196323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.196328 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.196945 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.197489 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.197495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.197500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.197678 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.198519 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.198527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.198794 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.198950 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.199341 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2089,"thread_start_us":1963,"threads_started":1}
I20260620 14:54:16.200642 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.181310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.21.159.254:32891)
0620 14:54:16.181536 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.181543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.183724 (+ 2181us) server_negotiation.cc:407] Connection header received
0620 14:54:16.183943 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.183948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.184040 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.184161 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.185366 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.186470 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.188563 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.189288 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.189431 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.192123 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.192131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.192137 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.192190 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.196131 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.197095 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.197179 (+ 84us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.197186 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.197343 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.197823 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.197830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.197836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.198354 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.198530 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.199298 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.199578 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:54:16.201895 28340 tablet_copy_service.cc:140] P 5c179d685f7f4ba68e8a0760536cbb73: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dc4a2c20c0a4469da2830f9ad5355817 ({username='oryx', principal='oryx/127.21.159.253@KRBTEST.COM'} at 127.0.0.1:37654)
I20260620 14:54:16.202273 28340 tablet_copy_service.cc:161] P 5c179d685f7f4ba68e8a0760536cbb73: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dc4a2c20c0a4469da2830f9ad5355817 at {username='oryx', principal='oryx/127.21.159.253@KRBTEST.COM'} at 127.0.0.1:37654: session id = dc4a2c20c0a4469da2830f9ad5355817-00000000000000000000000000000000
I20260620 14:54:16.206822 28340 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:54:16.209384 28783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:54:16.217389 28783 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: tablet copy: Starting download of 0 data blocks...
I20260620 14:54:16.217768 28783 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: tablet copy: Starting download of 1 WAL segments...
I20260620 14:54:16.221108 28783 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:54:16.226467 28340 tablet_copy_service.cc:342] P 5c179d685f7f4ba68e8a0760536cbb73: Request end of tablet copy session dc4a2c20c0a4469da2830f9ad5355817-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.159.253@KRBTEST.COM'} at 127.0.0.1:37654
I20260620 14:54:16.226727 28340 tablet_copy_service.cc:433] P 5c179d685f7f4ba68e8a0760536cbb73: ending tablet copy session dc4a2c20c0a4469da2830f9ad5355817-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dc4a2c20c0a4469da2830f9ad5355817
I20260620 14:54:16.240137 28783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:16.248208 28895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:16.249703 28896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:16.256886 28898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:16.257226 28783 server_base.cc:1034] running on GCE node
I20260620 14:54:16.257843 28783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:16.258706 28783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:16.259933 28783 hybrid_clock.cc:648] HybridClock initialized: now 1781967256259862 us; error 40 us; skew 500 ppm
I20260620 14:54:16.260198 28783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967256, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:16.262506 28783 init.cc:377] Logged in from keytab as oryx/127.21.159.253@KRBTEST.COM (short username oryx)
I20260620 14:54:16.263943 28783 webserver.cc:460] Webserver started at http://127.21.159.253:40035/ using document root <none> and password file <none>
I20260620 14:54:16.264389 28783 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:16.264472 28783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:16.267544 28783 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 14:54:16.269548 28905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:16.270057 28783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:54:16.270231 28783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
uuid: "dc4a2c20c0a4469da2830f9ad5355817"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "87fb4050ded8c9692d3d6388f282c695"
server_key_iv: "9a82f1a784ce687c0700b225e6dcb5b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:16.270433 28783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:16.290823 28783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:16.295540 28783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:16.326009 28783 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.253:35607
I20260620 14:54:16.326041 28956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.253:35607 every 8 connection(s)
I20260620 14:54:16.327234 28783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
I20260620 14:54:16.333321 28957 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:54:16.338167 28957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Bootstrap starting.
I20260620 14:54:16.356889 28957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:54:16.357491 28957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Bootstrap complete.
I20260620 14:54:16.358713 28957 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } }
I20260620 14:54:16.359014 28957 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dc4a2c20c0a4469da2830f9ad5355817, State: Initialized, Role: LEARNER
I20260620 14:54:16.359309 28957 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } }
I20260620 14:54:16.360364 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } }
I20260620 14:54:16.360653 28957 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:16.360832 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:54:16.374351 28971 catalog_manager.cc:1259] Loaded cluster ID: 4ef396ced08041b484a5914dc262644d
I20260620 14:54:16.374483 28971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: loading cluster ID for follower catalog manager: success
I20260620 14:54:16.378150 28971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: acquiring CA information for follower catalog manager: success
I20260620 14:54:16.380872 28971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:54:16.463227 28777 heartbeater.cc:503] Master 127.21.159.254:32891 was elected leader, sending a full tablet report...
I20260620 14:54:16.539222 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.528841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.0.0.1:53256)
0620 14:54:16.529143 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.529166 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.529361 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.530380 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.530385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.530425 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:16.530618 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.530629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.532892 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.532899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.534958 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.534976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.538320 (+ 3344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.538328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.538572 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.538658 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.538765 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.538921 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260620 14:54:16.539418 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.529072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53256 (local address 127.21.159.253:35607)
0620 14:54:16.529965 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.529973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.529994 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:54:16.530068 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.530073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.530173 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.530306 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:16.530798 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.532713 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.535181 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.538074 (+ 2893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.538340 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.538600 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.538820 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.538972 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":551,"threads_started":1}
I20260620 14:54:16.541013 28932 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 14:54:16.541683 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c179d685f7f4ba68e8a0760536cbb73. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } }
I20260620 14:54:16.542001 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:54:16.573639 28886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:16.574031 28886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:16.574184 28886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:16.581761 28886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:16.581943 28886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:16.582053 28886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:16.582154 28886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:16.582238 28886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:16.593048 28886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.252.keytab
--principal=oryx/127.21.159.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:39419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.159.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:16.594679 28886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:16.596215 28886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:16.609441 28977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:16.609408 28980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:16.609465 28978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:16.610308 28886 server_base.cc:1034] running on GCE node
I20260620 14:54:16.611239 28886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:16.612749 28886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:16.613963 28886 hybrid_clock.cc:648] HybridClock initialized: now 1781967256613922 us; error 44 us; skew 500 ppm
I20260620 14:54:16.614337 28886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967256, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:16.617451 28886 init.cc:377] Logged in from keytab as oryx/127.21.159.252@KRBTEST.COM (short username oryx)
I20260620 14:54:16.618927 28886 webserver.cc:460] Webserver started at http://127.21.159.252:33765/ using document root <none> and password file <none>
I20260620 14:54:16.619684 28886 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:16.619843 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:16.620186 28886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:16.623346 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/instance:
uuid: "9ff9d3e39cc647e5b400c285b5df56f2"
format_stamp: "Formatted at 2026-06-20 14:54:16 on dist-test-slave-3mqx"
server_key: "c8d3598bf776064373b2aea4d39d94a6"
server_key_iv: "5163177846af5d8efa302e3cb9905288"
server_key_version: "encryptionkey@0"
I20260620 14:54:16.624219 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal/instance:
uuid: "9ff9d3e39cc647e5b400c285b5df56f2"
format_stamp: "Formatted at 2026-06-20 14:54:16 on dist-test-slave-3mqx"
server_key: "c8d3598bf776064373b2aea4d39d94a6"
server_key_iv: "5163177846af5d8efa302e3cb9905288"
server_key_version: "encryptionkey@0"
I20260620 14:54:16.630048 28886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:54:16.633796 28987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:16.635402 28886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:16.635598 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
uuid: "9ff9d3e39cc647e5b400c285b5df56f2"
format_stamp: "Formatted at 2026-06-20 14:54:16 on dist-test-slave-3mqx"
server_key: "c8d3598bf776064373b2aea4d39d94a6"
server_key_iv: "5163177846af5d8efa302e3cb9905288"
server_key_version: "encryptionkey@0"
I20260620 14:54:16.635818 28886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:16.653822 28886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:16.658738 28886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:16.659044 28886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:16.680962 28886 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.252:39419
I20260620 14:54:16.680990 29038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.252:39419 every 8 connection(s)
I20260620 14:54:16.682534 28886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
I20260620 14:54:16.685369 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 28886
I20260620 14:54:16.687367 29039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:16.696578 29039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.252@KRBTEST.COM
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967256, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.252@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:16.711441 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.687366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.21.159.252:39419)
0620 14:54:16.688601 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.688613 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.688655 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:54:16.690221 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.690244 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.690660 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.691020 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.692920 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.694446 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.696346 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.697245 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.697639 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.703177 (+ 5538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.703186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.703197 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.703232 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.706805 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.707601 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.707609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.707617 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.707804 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.708153 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.708160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.708165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.708792 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.709050 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.709698 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.710152 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":362,"threads_started":1}
I20260620 14:54:16.716131 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.700094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.21.159.254:32891)
0620 14:54:16.700339 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.700385 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.700969 (+ 584us) server_negotiation.cc:407] Connection header received
0620 14:54:16.701263 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.701269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.701351 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.701478 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.702668 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.703584 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.705767 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.706146 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.706245 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.710046 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.710051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.710054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.710085 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.713188 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.714020 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.714030 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.714032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.714159 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.714650 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.714656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.714659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.714988 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.715274 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.715669 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.715845 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:54:16.716580 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.700001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:37660)
0620 14:54:16.700828 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.700873 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.701108 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.701569 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.701574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.702268 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.702515 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.702524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.703735 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.703741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.705607 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.705625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.706346 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.706355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.706564 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.707226 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.707242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.709860 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.713337 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.713347 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.713363 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.713876 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.714279 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.714284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.714288 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.714519 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.715201 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.715207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.715415 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.715582 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.715826 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":396,"threads_started":1}
I20260620 14:54:16.718014 29039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967256, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.252@KRBTEST.COM for oryx/127.21.159.253@KRBTEST.COM
I20260620 14:54:16.730595 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.718953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.21.159.253:35607)
0620 14:54:16.719197 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.719202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.719219 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:54:16.719329 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.719333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.719428 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.719509 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.720148 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.720873 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.722113 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.722506 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.722597 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.725590 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.725595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.725598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.725632 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.728307 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.728976 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.728980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.728982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.729098 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.729472 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.729475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.729477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.729737 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.729845 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.730181 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.730368 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:54:16.730631 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.718805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.0.0.1:53272)
0620 14:54:16.719008 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.719029 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.719223 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.719532 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.719535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.719881 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.720016 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.720024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.721006 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.721009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.721990 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.722001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.722639 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.722643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.722771 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.723382 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.723400 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.725398 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.728468 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.728472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.728475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.728868 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.729224 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.729228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.729231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.729373 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.729852 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.729855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.729983 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.730098 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.730390 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:54:16.731815 29039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967256, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.252@KRBTEST.COM for oryx/127.21.159.252@KRBTEST.COM
I20260620 14:54:16.745163 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.732540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.0.0.1:58116)
0620 14:54:16.732763 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.732779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.732895 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.733186 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.733190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.733580 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.733718 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.733725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.734674 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.734682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.735715 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.735727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.736445 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.736450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.736669 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.737263 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.737279 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.739388 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.742911 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.742914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.742917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.743343 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.743729 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.743732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.743734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.743890 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.744406 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.744409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.744539 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.744643 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.744833 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":22}
I20260620 14:54:16.745347 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.732780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.21.159.252:39419)
0620 14:54:16.732945 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.732953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.732971 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:54:16.733012 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.733017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.733075 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.733169 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.733847 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.734554 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.735865 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.736284 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.736446 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.739591 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.739596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.739600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.739630 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.742765 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.743476 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.743481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.743483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.743622 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.743988 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.743992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.743994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.744302 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.744410 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.744789 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.744978 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":58}
I20260620 14:54:16.750409 29039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Bootstrap starting.
I20260620 14:54:16.753818 29039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:16.754998 29039 log.cc:824] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:16.758013 29039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: No bootstrap required, opened a new log
I20260620 14:54:16.766628 29039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } }
I20260620 14:54:16.767026 29039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:16.767217 29039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ff9d3e39cc647e5b400c285b5df56f2, State: Initialized, Role: FOLLOWER
I20260620 14:54:16.767987 29039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } }
I20260620 14:54:16.769107 29045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } } }
I20260620 14:54:16.769441 29045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:54:16.770198 29039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:16.773084 29045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:16.773401 29045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } }
I20260620 14:54:16.774974 29045 leader_election.cc:288] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891), dc4a2c20c0a4469da2830f9ad5355817 (127.21.159.253:35607)
I20260620 14:54:16.775758 28932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc4a2c20c0a4469da2830f9ad5355817" is_pre_election: true
I20260620 14:54:16.775777 28330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c179d685f7f4ba68e8a0760536cbb73" is_pre_election: true
I20260620 14:54:16.776870 28991 leader_election.cc:302] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ff9d3e39cc647e5b400c285b5df56f2; no voters: 5c179d685f7f4ba68e8a0760536cbb73, dc4a2c20c0a4469da2830f9ad5355817
I20260620 14:54:16.777773 29045 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 14:54:16.781888 29056 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:54:16.782027 29056 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:54:16.785862 28886 master_runner.cc:428] Detected that this master 9ff9d3e39cc647e5b400c285b5df56f2 is joining an existing cluster
I20260620 14:54:16.785969 28886 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.159.252:39419
I20260620 14:54:16.810191 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.790341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.21.159.254:32891)
0620 14:54:16.790532 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.790538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.791296 (+ 758us) server_negotiation.cc:407] Connection header received
0620 14:54:16.791596 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.791605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.791711 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.791813 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.793875 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.795490 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.798508 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.799205 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.799312 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.802138 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.802144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.802151 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.802196 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.806290 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.807233 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.807239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.807242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.807792 (+ 550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.808297 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.808302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.808305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.808674 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.808798 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.809536 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.809763 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:54:16.811605 29062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.790395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:37668)
0620 14:54:16.791171 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.791195 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.791423 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.792091 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.792099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.793032 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.793649 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.793668 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.795783 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.795791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.797884 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.797899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.799596 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.799602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.799846 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.800611 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.800639 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.801900 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.806462 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.806467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.806472 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.807076 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.807946 (+ 870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.807955 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.807960 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.808164 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.808904 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.808909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.809176 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.810495 (+ 1319us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.811020 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":396,"threads_started":1}
I20260620 14:54:16.816067 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.790973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.0.0.1:53278)
0620 14:54:16.798629 (+ 7656us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.798650 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.798830 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.799387 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.799391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.800139 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.800540 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.800554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.802161 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.802165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.804055 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.804071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.805391 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.805397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.805632 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.806294 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.806319 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.808078 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.811915 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.811919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.811922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.812512 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.812940 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.812945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.812948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.813141 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.813918 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.813923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.814135 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.814911 (+ 776us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.815511 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7500,"mutex_wait_us":4,"spinlock_wait_cycles":1408,"thread_start_us":6633,"threads_started":1}
I20260620 14:54:16.818107 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.790428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.21.159.253:35607)
0620 14:54:16.790590 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.790596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.798727 (+ 8131us) server_negotiation.cc:407] Connection header received
0620 14:54:16.798966 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.798972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.799076 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.799153 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.800745 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.801896 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.804226 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.804969 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.805102 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.808293 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.808299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.808304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.808346 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.811741 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.812654 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.812660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.812664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.812805 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.813286 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.813292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.813295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.813661 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.813814 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.814431 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.817772 (+ 3341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:54:16.818300 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 28381
I20260620 14:54:16.824208 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.791341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.21.159.252:39419)
0620 14:54:16.791507 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.791514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.804138 (+ 12624us) server_negotiation.cc:407] Connection header received
0620 14:54:16.804330 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.804337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.804407 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.804495 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.805905 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.807077 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.809492 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.810231 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.810326 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.813200 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.813206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.813211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.813249 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.817150 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.821300 (+ 4150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.821308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.821315 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.821530 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.822006 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.822012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.822021 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.822505 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.822693 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.823345 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.823669 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:54:16.825886 29064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.791482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.0.0.1:58130)
0620 14:54:16.804012 (+ 12530us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.804030 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.804204 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.804557 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.804562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.805300 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.805733 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.805746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.807701 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.807707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.809345 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.809359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.810748 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.810756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.810978 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.811653 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.811810 (+ 157us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.813008 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.820428 (+ 7420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.820436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.820444 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.821139 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.821700 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.821707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.821713 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.821879 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.822773 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.822780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.822999 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.825083 (+ 2084us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.825403 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12389,"thread_start_us":12221,"threads_started":1}
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.254@KRBTEST.COM for oryx/127.21.159.252@KRBTEST.COM
I20260620 14:54:16.844987 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.817868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.0.0.1:58134)
0620 14:54:16.818263 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.818283 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.818396 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.824969 (+ 6573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.824974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.825469 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.825727 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.825740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.829770 (+ 4030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.829779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.831837 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.831862 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.833432 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.833442 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.833702 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.834760 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.834795 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.837559 (+ 2764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.841948 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.841956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.841962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.842484 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.842979 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.842984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.842988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.843208 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.843924 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.843929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.844103 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.844244 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.844572 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260620 14:54:16.846185 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.818594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.21.159.252:39419)
0620 14:54:16.824512 (+ 5918us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.824520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.824543 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:54:16.824662 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.824670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.824786 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.824914 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.825934 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.829539 (+ 3605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.832154 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.833155 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.835812 (+ 2657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.837776 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.837787 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.837791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.837825 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.841720 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.842633 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.842639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.842643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.842822 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.843350 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.843355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.843358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.843705 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.843850 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.845543 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.845803 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5771,"thread_start_us":17050,"threads_started":1}
Jun 20 14:54:16 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.193@KRBTEST.COM
I20260620 14:54:16.847926 28320 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.159.252:39419: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2"
member_type: NON_VOTER
last_known_addr {
host: "127.21.159.252"
port: 39419
}
attrs {
promote: true
}
}
dest_uuid: "5c179d685f7f4ba68e8a0760536cbb73"
cas_config_opid_index: 5
I20260620 14:54:16.848816 28320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } }
I20260620 14:54:16.850878 28881 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.856103 28932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 LEARNER]: Refusing update from remote peer 5c179d685f7f4ba68e8a0760536cbb73: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:54:16.856377 29014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:16.857877 29066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.820482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.21.159.193:35161)
0620 14:54:16.835959 (+ 15477us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.835969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.836005 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:54:16.836080 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.836089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.836930 (+ 841us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.838507 (+ 1577us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.838875 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.840433 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.841811 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.842779 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.842921 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.848032 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.848052 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.848074 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.848128 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.853102 (+ 4974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.854254 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.854267 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.854271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.854447 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.854963 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.854969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.854972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.855454 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.855758 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.857108 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.857433 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14979,"thread_start_us":15111,"threads_started":1}
I20260620 14:54:16.859542 28504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:16.870797 29014 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Refusing update from remote peer 5c179d685f7f4ba68e8a0760536cbb73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:54:16.874766 29070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260620 14:54:16.876744 28881 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.159.252:39419
I20260620 14:54:16.879442 28886 master.cc:537] Master@127.21.159.252:39419 shutting down...
I20260620 14:54:16.888991 29070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:16.894224 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.895944 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:16.895388 28932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.900791 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: NON_VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: true } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.901129 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:54:16.902495 29068 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: attempting to promote NON_VOTER dc4a2c20c0a4469da2830f9ad5355817 to VOTER
I20260620 14:54:16.903596 29068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } }
I20260620 14:54:16.907248 28931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 LEARNER]: Refusing update from remote peer 5c179d685f7f4ba68e8a0760536cbb73: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:54:16.907981 29070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:54:16.910677 29068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc4a2c20c0a4469da2830f9ad5355817 (127.21.159.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
W20260620 14:54:16.916198 28993 tablet.cc:2369] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:16.916394 28931 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dc4a2c20c0a4469da2830f9ad5355817 (127.21.159.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.917398 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.917892 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:16.918215 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.918421 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:16.921756 28886 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:16.922191 28886 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:16.922377 28886 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: stopping tablet replica
I20260620 14:54:16.922730 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:16.922992 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:54:16.926793 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.926333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.0.0.1:58144)
0620 14:54:16.926484 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.926602 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:39419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:54:16.927139 28304 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 -> Peer 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252:39419): Couldn't send request to peer 9ff9d3e39cc647e5b400c285b5df56f2. Status: Network error: Client connection negotiation failed: client connection to 127.21.159.252:39419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:54:16.929651 28886 master.cc:552] Master@127.21.159.252:39419 shutdown complete.
I20260620 14:54:16.929741 28886 master_runner.cc:305] Clearing existing system tablet
I20260620 14:54:16.930130 28886 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:16.930241 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:16.933324 28886 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 14:54:16.935081 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:16.935437 28886 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:16.935873 28886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 14:54:16.936136 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
uuid: "9ff9d3e39cc647e5b400c285b5df56f2"
format_stamp: "Formatted at 2026-06-20 14:54:16 on dist-test-slave-3mqx"
server_key: "c8d3598bf776064373b2aea4d39d94a6"
server_key_iv: "5163177846af5d8efa302e3cb9905288"
server_key_version: "encryptionkey@0"
I20260620 14:54:16.941584 28886 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:16.949162 28886 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:54:16.949489 28886 log.cc:1196] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:54:16.950156 28886 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Deleting consensus metadata
I20260620 14:54:16.950485 28886 master_runner.cc:315] Copying system tablet from 127.21.159.254:32891
I20260620 14:54:16.953454 28886 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: tablet copy: Beginning tablet copy session from remote peer at address 127.21.159.254:32891
I20260620 14:54:16.970782 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.954538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.21.159.254:32891)
0620 14:54:16.954793 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:16.954803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:16.955241 (+ 438us) server_negotiation.cc:407] Connection header received
0620 14:54:16.955505 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:16.955511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:16.955607 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:16.955674 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:16.956931 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.958081 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.960019 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.960702 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.960803 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.963571 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:16.963578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:16.963584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:16.963624 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:16.967255 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.968183 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.968190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.968194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.968351 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:16.968942 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:16.968957 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:16.968961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:16.969341 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:16.969485 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:16.970029 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 14:54:16.970266 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 14:54:16.971170 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:16.954531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:37682)
0620 14:54:16.955124 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:16.955146 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:16.955362 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:16.955927 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:16.955932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:16.956458 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:16.956719 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.956733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.958277 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.958283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.959839 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:16.959855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:16.961047 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:16.961053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:16.961247 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:16.962083 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:16.962108 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:16.963326 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:16.967449 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.967455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.967459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.968012 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.968527 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:16.968532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:16.968536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:16.968716 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:16.969473 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:16.969479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:16.969674 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 14:54:16.969826 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:54:16.970175 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":270,"threads_started":1}
I20260620 14:54:16.971973 28340 tablet_copy_service.cc:140] P 5c179d685f7f4ba68e8a0760536cbb73: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9ff9d3e39cc647e5b400c285b5df56f2 ({username='oryx', principal='oryx/127.21.159.252@KRBTEST.COM'} at 127.0.0.1:37682)
I20260620 14:54:16.972138 28340 tablet_copy_service.cc:161] P 5c179d685f7f4ba68e8a0760536cbb73: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9ff9d3e39cc647e5b400c285b5df56f2 at {username='oryx', principal='oryx/127.21.159.252@KRBTEST.COM'} at 127.0.0.1:37682: session id = 9ff9d3e39cc647e5b400c285b5df56f2-00000000000000000000000000000000
I20260620 14:54:16.975879 28340 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:54:16.977864 28886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:54:16.985235 28886 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: tablet copy: Starting download of 0 data blocks...
I20260620 14:54:16.985580 28886 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: tablet copy: Starting download of 1 WAL segments...
I20260620 14:54:16.988294 28886 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:54:16.993182 28340 tablet_copy_service.cc:342] P 5c179d685f7f4ba68e8a0760536cbb73: Request end of tablet copy session 9ff9d3e39cc647e5b400c285b5df56f2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.159.252@KRBTEST.COM'} at 127.0.0.1:37682
I20260620 14:54:16.993336 28340 tablet_copy_service.cc:433] P 5c179d685f7f4ba68e8a0760536cbb73: ending tablet copy session 9ff9d3e39cc647e5b400c285b5df56f2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9ff9d3e39cc647e5b400c285b5df56f2
I20260620 14:54:17.006208 28886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:17.009634 29082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:17.011407 29083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:17.018853 29085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:17.025120 28886 server_base.cc:1034] running on GCE node
I20260620 14:54:17.025712 28886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:17.026439 28886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:17.027034 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 28381
I20260620 14:54:17.027727 28886 hybrid_clock.cc:648] HybridClock initialized: now 1781967257027675 us; error 50 us; skew 500 ppm
I20260620 14:54:17.028050 28886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:17 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:17.030346 28886 init.cc:377] Logged in from keytab as oryx/127.21.159.252@KRBTEST.COM (short username oryx)
I20260620 14:54:17.044256 28886 webserver.cc:460] Webserver started at http://127.21.159.252:42675/ using document root <none> and password file <none>
I20260620 14:54:17.044677 28886 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:17.044785 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:17.046167 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:35161
--local_ip_for_outbound_sockets=127.21.159.193
--tserver_master_addrs=127.21.159.254:32891
--webserver_port=34189
--webserver_interface=127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab
--principal=oryx/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:54:17.048130 28886 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 14:54:17.050220 29092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:17.050711 28886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:54:17.050894 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
uuid: "9ff9d3e39cc647e5b400c285b5df56f2"
format_stamp: "Formatted at 2026-06-20 14:54:16 on dist-test-slave-3mqx"
server_key: "c8d3598bf776064373b2aea4d39d94a6"
server_key_iv: "5163177846af5d8efa302e3cb9905288"
server_key_version: "encryptionkey@0"
I20260620 14:54:17.051229 28886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:17.074759 28886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:17.079226 28886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:17.109290 28886 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.252:39419
I20260620 14:54:17.109333 29144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.252:39419 every 8 connection(s)
I20260620 14:54:17.110294 28886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
I20260620 14:54:17.115763 29145 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:54:17.119968 29145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Bootstrap starting.
I20260620 14:54:17.140300 29145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:54:17.140921 29145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: Bootstrap complete.
I20260620 14:54:17.142144 29145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } }
I20260620 14:54:17.142462 29145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ff9d3e39cc647e5b400c285b5df56f2, State: Initialized, Role: LEARNER
I20260620 14:54:17.142757 29145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } }
I20260620 14:54:17.143784 29148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:17.144296 29145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:17.144296 29148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:54:17.157158 29159 catalog_manager.cc:1259] Loaded cluster ID: 4ef396ced08041b484a5914dc262644d
I20260620 14:54:17.157292 29159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: loading cluster ID for follower catalog manager: success
I20260620 14:54:17.160857 29159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: acquiring CA information for follower catalog manager: success
I20260620 14:54:17.163616 29159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:54:17.377614 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.369625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.21.159.252:39419)
0620 14:54:17.370269 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:17.370274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:17.370292 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:54:17.370357 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:17.370361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:17.370439 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:17.370546 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:17.371145 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.372304 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.375461 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.376360 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.376551 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.376706 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:17.377160 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 14:54:17.377286 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":371,"threads_started":1}
I20260620 14:54:17.377664 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.369334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.0.0.1:58154)
0620 14:54:17.369548 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:17.369574 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:17.369733 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:17.370621 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:17.370627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:17.370681 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:17.370916 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.370930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.372477 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.372486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.375217 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.375241 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.376602 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.376612 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.376886 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.376938 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:54:17.377098 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 14:54:17.377285 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:54:17.379259 29120 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:54:17.380070 29148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c179d685f7f4ba68e8a0760536cbb73. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:17.380440 29148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: This master's current role is: LEARNER
W20260620 14:54:17.446885 29093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:17.447362 29093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:17.447570 29093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:17.456198 29093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:17.456353 29093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:17.456565 29093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:17.467792 29093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab
--principal=oryx/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:35161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=34189
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:17.469310 29093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:17.470824 29093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:17.486490 29168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:17.486568 29165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:17.486526 29166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:17.487448 29093 server_base.cc:1034] running on GCE node
I20260620 14:54:17.488126 29093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:17.489243 29093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:17.490494 29093 hybrid_clock.cc:648] HybridClock initialized: now 1781967257490419 us; error 94 us; skew 500 ppm
I20260620 14:54:17.490906 29093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:17 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:17.494845 29093 init.cc:377] Logged in from keytab as oryx/127.21.159.193@KRBTEST.COM (short username oryx)
I20260620 14:54:17.496708 29093 webserver.cc:460] Webserver started at http://127.21.159.193:34189/ using document root <none> and password file <none>
I20260620 14:54:17.497502 29093 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:17.497669 29093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:17.504491 29093 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:54:17.507588 29175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:17.509099 29093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:54:17.509310 29093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "0613f1ae65d84469b92626f98ee51cc2"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "0c1b5038484a3ac8fbc87a543098f9e0"
server_key_iv: "ef66d9ce1e426d933142915b48d44e32"
server_key_version: "encryptionkey@0"
I20260620 14:54:17.510428 29093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:17.537654 29093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:17.542580 29093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:17.542924 29093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:17.544094 29093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:17.546128 29093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:17.546271 29093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:17.546452 29093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:17.546564 29093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:17.585603 29093 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:35161
I20260620 14:54:17.585695 29287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:35161 every 8 connection(s)
I20260620 14:54:17.587247 29093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:17.591732 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 29093
I20260620 14:54:17.592160 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 28515
Jun 20 14:54:17 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.253@KRBTEST.COM
Jun 20 14:54:17 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
Jun 20 14:54:17 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.194@KRBTEST.COM
Jun 20 14:54:17 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.252@KRBTEST.COM
I20260620 14:54:17.624778 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.590168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:48847 (local address 127.21.159.253:35607)
0620 14:54:17.590830 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:17.590840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:17.593903 (+ 3063us) server_negotiation.cc:407] Connection header received
0620 14:54:17.599989 (+ 6086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:17.599999 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:17.600157 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:17.600265 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:17.603581 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.604791 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.607794 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.608480 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.608599 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.616198 (+ 7599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:17.616206 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:17.616212 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:17.616258 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:17.620669 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.621540 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.621548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.621552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.621700 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.622339 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.622344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.622347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.622715 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:17.622855 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:17.624128 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0620 14:54:17.624389 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":420,"threads_started":1}
I20260620 14:54:17.626907 29294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.591150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.21.159.193:48847)
0620 14:54:17.593699 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:17.593758 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:17.598448 (+ 4690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:17.601123 (+ 2675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:17.601141 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:17.602201 (+ 1060us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:17.603315 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.603349 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.605023 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.605028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.607622 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.607640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.608959 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.608966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.609328 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.612116 (+ 2788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:17.612150 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:17.615859 (+ 3709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:17.620897 (+ 5038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.620908 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.620929 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.621380 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.621852 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.621861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.621865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.622178 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.622971 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:17.622983 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:17.623596 (+ 613us) client_negotiation.cc:769] Sending connection context
0620 14:54:17.625010 (+ 1414us) client_negotiation.cc:240] Negotiation successful
0620 14:54:17.625462 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2161,"mutex_wait_us":4,"thread_start_us":1877,"threads_started":1}
I20260620 14:54:17.627434 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.590502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:50919 (local address 127.21.159.254:32891)
0620 14:54:17.603032 (+ 12530us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:17.603042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:17.605626 (+ 2584us) server_negotiation.cc:407] Connection header received
0620 14:54:17.605703 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:17.605709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:17.605796 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:17.605897 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:17.607586 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.609043 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.613001 (+ 3958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.613834 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.613979 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.618691 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:17.618703 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:17.618710 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:17.618776 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:17.622526 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.623739 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.623746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.623750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.623896 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.624349 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.624354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.624357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.624706 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:17.624869 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:17.626776 (+ 1907us) server_negotiation.cc:299] Negotiation successful
0620 14:54:17.627028 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12348,"thread_start_us":12571,"threads_started":1}
I20260620 14:54:17.627574 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.591820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.193:50919)
0620 14:54:17.605332 (+ 13512us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:17.605358 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:17.605502 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:17.606098 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:17.606103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:17.606921 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:17.607375 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.607390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.609261 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.609270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.612807 (+ 3537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.612822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.614322 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.614328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.614498 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.615212 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:17.615235 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:17.618414 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:17.622867 (+ 4453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.622874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.622883 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.623564 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.624040 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.624045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.624050 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.624203 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.626267 (+ 2064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:17.626272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:17.626452 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 14:54:17.626632 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 14:54:17.626955 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13367,"mutex_wait_us":393,"thread_start_us":13298,"threads_started":1}
I20260620 14:54:17.629032 29290 heartbeater.cc:347] Connected to a master server at 127.21.159.253:35607
I20260620 14:54:17.629185 29289 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32891
I20260620 14:54:17.629652 29289 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:17.629698 29290 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:17.630864 29289 heartbeater.cc:511] Master 127.21.159.254:32891 requested a full tablet report, sending...
I20260620 14:54:17.630889 29290 heartbeater.cc:511] Master 127.21.159.253:35607 requested a full tablet report, sending...
I20260620 14:54:17.631562 29297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.593839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.21.159.194:41545)
0620 14:54:17.603813 (+ 9974us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:17.603826 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:17.604761 (+ 935us) server_negotiation.cc:407] Connection header received
0620 14:54:17.604855 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:17.604865 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:17.605172 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:17.605362 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:17.607774 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.609311 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.611761 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.612704 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.612831 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.621862 (+ 9031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:17.621884 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:17.621905 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:17.621951 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:17.626867 (+ 4916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.628116 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.628129 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.628136 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.628319 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.628799 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.628804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.628807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.629280 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:17.629578 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:17.630811 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0620 14:54:17.631127 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9752,"thread_start_us":9527,"threads_started":1}
I20260620 14:54:17.631980 29160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.605807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:59483 (local address 127.21.159.252:39419)
0620 14:54:17.606005 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:17.606010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:17.611617 (+ 5607us) server_negotiation.cc:407] Connection header received
0620 14:54:17.612283 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:17.612292 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:17.612383 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:17.612462 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:17.613919 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.615105 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.617145 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.617856 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.618389 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.622417 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:17.622426 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:17.622432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:17.622472 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:17.625958 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.628234 (+ 2276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.628242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.628245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.628408 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.629150 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.629155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.629158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.629975 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:17.630129 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:17.631354 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0620 14:54:17.631630 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:54:17.632891 29298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.606157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.21.159.193:59483)
0620 14:54:17.611473 (+ 5316us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:17.611498 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:17.612103 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:17.612715 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:17.612719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:17.613309 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:17.613710 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.613725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.615272 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.615281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.617016 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.617035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.618686 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.618695 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.618901 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.619502 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:17.619529 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:17.622192 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:17.626114 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.626119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.626122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.628091 (+ 1969us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.628802 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.628807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.628810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.628979 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.630864 (+ 1885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:17.630868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:17.631038 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:54:17.632220 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0620 14:54:17.632551 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5076,"thread_start_us":3334,"threads_started":1}
I20260620 14:54:17.632899 28320 ts_manager.cc:194] Re-registered known tserver with Master: 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161)
I20260620 14:54:17.633517 28922 ts_manager.cc:194] Registered new tserver with Master: 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161)
I20260620 14:54:17.633906 29288 heartbeater.cc:347] Connected to a master server at 127.21.159.252:39419
I20260620 14:54:17.633991 28638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:17.634049 29288 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:17.634039 28320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:50919
I20260620 14:54:17.634698 29288 heartbeater.cc:511] Master 127.21.159.252:39419 requested a full tablet report, sending...
I20260620 14:54:17.637219 29110 ts_manager.cc:194] Registered new tserver with Master: 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161)
I20260620 14:54:17.719431 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 28515
I20260620 14:54:17.736485 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:41545
--local_ip_for_outbound_sockets=127.21.159.194
--tserver_master_addrs=127.21.159.254:32891
--webserver_port=40481
--webserver_interface=127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab
--principal=oryx/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:54:17.969849 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: NON_VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: true } } }
I20260620 14:54:17.970191 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:17.979867 29302 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: attempting to promote NON_VOTER 9ff9d3e39cc647e5b400c285b5df56f2 to VOTER
I20260620 14:54:17.980674 29302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } }
I20260620 14:54:17.983428 28931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Refusing update from remote peer 5c179d685f7f4ba68e8a0760536cbb73: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 14:54:17.983899 29120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 LEARNER]: Refusing update from remote peer 5c179d685f7f4ba68e8a0760536cbb73: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 14:54:17.984100 29301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:54:17.986004 29301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:54:17.988372 29301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:17.992079 28931 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:18.000866 29302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:18.001240 29302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:18.001765 29302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:18.002293 29302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:18.003507 29307 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:18.003966 29307 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:54:18.004714 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:18.005018 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:18.005777 29120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:18.012535 29308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c179d685f7f4ba68e8a0760536cbb73" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:18.012933 29308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:54:18.098078 29300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:18.098374 29300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:18.098565 29300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:18.106609 29300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:18.106734 29300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:18.106916 29300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:18.117545 29300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab
--principal=oryx/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:41545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=40481
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:18.118983 29300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:18.120440 29300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:18.138398 29314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:18.138677 29313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:18.138952 29316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:18.139654 29300 server_base.cc:1034] running on GCE node
I20260620 14:54:18.140316 29300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:18.141465 29300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:18.142793 29300 hybrid_clock.cc:648] HybridClock initialized: now 1781967258142713 us; error 72 us; skew 500 ppm
I20260620 14:54:18.143282 29300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:18.146957 29300 init.cc:377] Logged in from keytab as oryx/127.21.159.194@KRBTEST.COM (short username oryx)
I20260620 14:54:18.148694 29300 webserver.cc:460] Webserver started at http://127.21.159.194:40481/ using document root <none> and password file <none>
I20260620 14:54:18.149363 29300 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:18.149475 29300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:18.155362 29300 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:54:18.158334 29323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.159904 29300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:18.160099 29300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "bb466bdeae9b45daa55de809056af60e"
format_stamp: "Formatted at 2026-06-20 14:54:14 on dist-test-slave-3mqx"
server_key: "609f74381a55a39acca214169059e7db"
server_key_iv: "f0edb850b9b03d8a715e5afe4d43c743"
server_key_version: "encryptionkey@0"
I20260620 14:54:18.161178 29300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:18.181696 29300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:18.186604 29300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:18.186921 29300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:18.188015 29300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:18.189468 29300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:18.189599 29300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.189738 29300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:18.189827 29300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.231756 29300 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:41545
I20260620 14:54:18.231810 29435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:41545 every 8 connection(s)
I20260620 14:54:18.233265 29300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:18.235311 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 29300
I20260620 14:54:18.235661 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 28649
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.195@KRBTEST.COM
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.252@KRBTEST.COM
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.253@KRBTEST.COM
I20260620 14:54:18.293152 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.236634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.21.159.195:43335)
0620 14:54:18.262760 (+ 26126us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.262776 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.262823 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:54:18.262925 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.262939 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.263438 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.263667 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.269956 (+ 6289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.271342 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.273878 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.275116 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.275301 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.283582 (+ 8281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.283605 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.283624 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.283676 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.289187 (+ 5511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.290224 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.290235 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.290239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.290415 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.290913 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.290922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.290925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.291427 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.292029 (+ 602us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.292396 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.292687 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25843,"thread_start_us":27332,"threads_started":1}
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:18.297024 28772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:18.302560 29445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.243721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:43383 (local address 127.21.159.253:35607)
0620 14:54:18.275533 (+ 31812us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.275540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.275812 (+ 272us) server_negotiation.cc:407] Connection header received
0620 14:54:18.276942 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.276949 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.277052 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.277168 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.280640 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.281960 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.285995 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.286741 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.286857 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.293490 (+ 6633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.293499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.293513 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.293586 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.297756 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.298613 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.298618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.298621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.298753 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.299363 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.299373 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.299380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.299981 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.300180 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.301588 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.301954 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31654,"thread_start_us":31808,"threads_started":1}
I20260620 14:54:18.303956 29442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.238739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:57315 (local address 127.21.159.252:39419)
0620 14:54:18.275973 (+ 37234us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.275980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.276004 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:54:18.276870 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.276876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.276969 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.277085 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.281244 (+ 4159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.282460 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.285993 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.286750 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.286860 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.293332 (+ 6472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.293344 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.293352 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.293409 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.299236 (+ 5827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.300803 (+ 1567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.300812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.300817 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.301431 (+ 614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.301967 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.301973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.301978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.302416 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.302615 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.303202 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.303470 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37105,"thread_start_us":37258,"threads_started":1}
I20260620 14:54:18.305428 29444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.237959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.21.159.194:57315)
0620 14:54:18.272993 (+ 35034us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.273059 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.276422 (+ 3363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.277675 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.277695 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.279084 (+ 1389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.281036 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.281055 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.282703 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.282708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.285808 (+ 3100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.285818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.287272 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.287281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.287751 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.288886 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.288944 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.292969 (+ 4025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.299978 (+ 7009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.299988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.299992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.300615 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.301633 (+ 1018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.301639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.301643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.301807 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.302586 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.302592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.302807 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.303085 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.303815 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34653,"mutex_wait_us":1485,"spinlock_wait_cycles":34048,"thread_start_us":38721,"threads_started":1}
I20260620 14:54:18.306550 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.243604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.21.159.194:43383)
0620 14:54:18.275600 (+ 31996us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.275628 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.276445 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.277682 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.277694 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.279043 (+ 1349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.280413 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.280446 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.282200 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.282210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.285791 (+ 3581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.285807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.287272 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.287281 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.287753 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.288886 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.288944 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.293221 (+ 4277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.297915 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.297925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.297946 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.298450 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.298877 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.298883 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.298887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.299174 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.300326 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.300340 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.300986 (+ 646us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.304025 (+ 3039us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.304634 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31831,"mutex_wait_us":280,"spinlock_wait_cycles":24192,"thread_start_us":28859,"threads_started":1}
I20260620 14:54:18.307436 29436 heartbeater.cc:347] Connected to a master server at 127.21.159.252:39419
I20260620 14:54:18.308611 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.277495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:57559 (local address 127.21.159.254:32891)
0620 14:54:18.280967 (+ 3472us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.280974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.283830 (+ 2856us) server_negotiation.cc:407] Connection header received
0620 14:54:18.284118 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.284125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.284217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.284306 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.286394 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.287708 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.290058 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.290792 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.290900 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.296940 (+ 6040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.296949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.296954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.297034 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.301420 (+ 4386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.304373 (+ 2953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.304381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.304385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.304576 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.305939 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.305946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.305949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.306377 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.306532 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.308082 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.308278 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3345,"thread_start_us":3231,"threads_started":1}
I20260620 14:54:18.308914 29436 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.309630 29438 heartbeater.cc:347] Connected to a master server at 127.21.159.253:35607
I20260620 14:54:18.309806 29438 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.310070 29436 heartbeater.cc:511] Master 127.21.159.252:39419 requested a full tablet report, sending...
I20260620 14:54:18.310832 29438 heartbeater.cc:511] Master 127.21.159.253:35607 requested a full tablet report, sending...
I20260620 14:54:18.308449 29447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.280898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:57559)
0620 14:54:18.283712 (+ 2814us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.283740 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.283946 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.284563 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.284569 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.285820 (+ 1251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.286178 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.286194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.287894 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.287902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.289863 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.289880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.291241 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.291248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.291438 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.293136 (+ 1698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.293165 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.296669 (+ 3504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.303100 (+ 6431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.303109 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.303117 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.303865 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.305305 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.305315 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.305322 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.305757 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.306876 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.306885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.307165 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.307350 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.307806 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2636,"spinlock_wait_cycles":6912,"thread_start_us":2378,"threads_started":1}
I20260620 14:54:18.312525 28922 ts_manager.cc:194] Registered new tserver with Master: bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545)
I20260620 14:54:18.312645 29110 ts_manager.cc:194] Registered new tserver with Master: bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545)
I20260620 14:54:18.313333 29437 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32891
I20260620 14:54:18.313494 29437 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.314086 29437 heartbeater.cc:511] Master 127.21.159.254:32891 requested a full tablet report, sending...
I20260620 14:54:18.315523 28320 ts_manager.cc:194] Re-registered known tserver with Master: bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545)
I20260620 14:54:18.317165 28320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:57559
I20260620 14:54:18.403469 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 28649
I20260620 14:54:18.420392 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:43335
--local_ip_for_outbound_sockets=127.21.159.195
--tserver_master_addrs=127.21.159.254:32891
--webserver_port=39029
--webserver_interface=127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab
--principal=oryx/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:54:18.637542 29289 heartbeater.cc:503] Master 127.21.159.254:32891 was elected leader, sending a full tablet report...
W20260620 14:54:18.772141 29449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:18.772516 29449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:18.772724 29449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:18.781037 29449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:18.781199 29449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:18.781373 29449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:18.791832 29449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab
--principal=oryx/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:43335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=39029
--tserver_master_addrs=127.21.159.254:32891,127.21.159.253:35607,127.21.159.252:39419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:18.793274 29449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:18.794859 29449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:18.810096 29455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:18.810061 29454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:18.810314 29457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:18.810935 29449 server_base.cc:1034] running on GCE node
I20260620 14:54:18.811605 29449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:18.812674 29449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:18.813931 29449 hybrid_clock.cc:648] HybridClock initialized: now 1781967258813862 us; error 63 us; skew 500 ppm
I20260620 14:54:18.814289 29449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:18.817803 29449 init.cc:377] Logged in from keytab as oryx/127.21.159.195@KRBTEST.COM (short username oryx)
I20260620 14:54:18.819422 29449 webserver.cc:460] Webserver started at http://127.21.159.195:39029/ using document root <none> and password file <none>
I20260620 14:54:18.820175 29449 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:18.820346 29449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:18.826262 29449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:54:18.829452 29464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.831228 29449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:18.831473 29449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "687f2d289d8d44a884e8c417758b1413"
format_stamp: "Formatted at 2026-06-20 14:54:15 on dist-test-slave-3mqx"
server_key: "d05a180e5b95064ade60543a4d85d938"
server_key_iv: "216da50db637446cd59653aa7ea94cc5"
server_key_version: "encryptionkey@0"
I20260620 14:54:18.832614 29449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:18.856452 29449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:18.860961 29449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:18.861366 29449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:18.862553 29449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:18.864424 29449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:18.864562 29449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.864703 29449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:18.864795 29449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.901089 29449 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:43335
I20260620 14:54:18.901176 29576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:43335 every 8 connection(s)
I20260620 14:54:18.902779 29449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:18.913316 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 29449
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.252@KRBTEST.COM
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.253@KRBTEST.COM
I20260620 14:54:18.957106 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.907273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.21.159.195:40603)
0620 14:54:18.930038 (+ 22765us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.930129 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.933451 (+ 3322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.934622 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.934636 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.935595 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.936544 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.936575 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.938669 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.938678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.942006 (+ 3328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.942030 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.943628 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.943636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.944029 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.944942 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.944966 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.947606 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.951152 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.951162 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.951182 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.951814 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.952268 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.952273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.952276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.952543 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.953487 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.953498 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.954452 (+ 954us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.954805 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.955504 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22250,"mutex_wait_us":50,"thread_start_us":28497,"threads_started":1}
I20260620 14:54:18.958652 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.905992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:40603 (local address 127.21.159.252:39419)
0620 14:54:18.921434 (+ 15442us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.921444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.933974 (+ 12530us) server_negotiation.cc:407] Connection header received
0620 14:54:18.934040 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.934044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.934124 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.934272 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.936872 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.938407 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.942256 (+ 3849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.943170 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.943275 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.947887 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.947894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.947899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.947942 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.950947 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.951933 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.951939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.951944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.952072 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.952744 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.952750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.952754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.953225 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.953401 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.957985 (+ 4584us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.958266 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15265,"thread_start_us":15454,"threads_started":1}
I20260620 14:54:18.960291 29577 heartbeater.cc:347] Connected to a master server at 127.21.159.252:39419
I20260620 14:54:18.960747 29577 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.961305 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.907656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:54483 (local address 127.21.159.253:35607)
0620 14:54:18.923854 (+ 16198us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.923863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.935865 (+ 12002us) server_negotiation.cc:407] Connection header received
0620 14:54:18.936020 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.936028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.936150 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.936290 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.937990 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.939502 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.944835 (+ 5333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.945736 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.945867 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.952366 (+ 6499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.952372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.952377 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.952438 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.956669 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.958028 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.958034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.958037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.958201 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.958632 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.958638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.958643 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.959903 (+ 1260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.960035 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.960625 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.960926 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16029,"thread_start_us":15925,"threads_started":1}
I20260620 14:54:18.961319 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.907273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.21.159.195:54483)
0620 14:54:18.935734 (+ 28461us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.935756 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.935896 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.936532 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.936538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.937284 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.937737 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.937755 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.939756 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.939764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.944143 (+ 4379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.944163 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.946249 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.946259 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.946543 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.947814 (+ 1271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.947843 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.952124 (+ 4281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.956874 (+ 4750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.956879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.956885 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.957850 (+ 965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.958356 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.958361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.958365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.958508 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.960115 (+ 1607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.960120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.960293 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.960521 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.960863 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28267,"mutex_wait_us":2517,"thread_start_us":26644,"threads_started":1}
I20260620 14:54:18.961874 29577 heartbeater.cc:511] Master 127.21.159.252:39419 requested a full tablet report, sending...
I20260620 14:54:18.962658 29579 heartbeater.cc:347] Connected to a master server at 127.21.159.253:35607
I20260620 14:54:18.962798 29579 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.963328 29579 heartbeater.cc:511] Master 127.21.159.253:35607 requested a full tablet report, sending...
I20260620 14:54:18.965500 29110 ts_manager.cc:194] Registered new tserver with Master: 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335)
I20260620 14:54:18.966756 28922 ts_manager.cc:194] Registered new tserver with Master: 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335)
Jun 20 14:54:18 dist-test-slave-3mqx krb5kdc[28275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:18.987373 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.926543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.21.159.253:35607)
0620 14:54:18.961994 (+ 35451us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.962001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.967590 (+ 5589us) server_negotiation.cc:407] Connection header received
0620 14:54:18.967865 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.967871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.967972 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.971068 (+ 3096us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.971236 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.972367 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.974310 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.974754 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.974909 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.979705 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.979712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.979717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.979761 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.983193 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.984085 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.984092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.984096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.984264 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.984787 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.984792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.984795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.985170 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.985471 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.986698 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.986943 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35309,"thread_start_us":38884,"threads_started":1}
I20260620 14:54:18.989181 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.967843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.21.159.252:39419)
0620 14:54:18.968058 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.968064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.968459 (+ 395us) server_negotiation.cc:407] Connection header received
0620 14:54:18.968651 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.968657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.968727 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.968818 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.971871 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.972953 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.974896 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.975584 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.976282 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.980824 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.980838 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.980842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.980877 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.984800 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.985532 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.985537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.985540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.985697 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.986139 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.986144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.986148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.986537 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.986779 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.988630 (+ 1851us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.988858 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:54:18.989464 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.936585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.195:42267)
0620 14:54:18.943352 (+ 6767us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.943374 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.943492 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.967402 (+ 23910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.967406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.968517 (+ 1111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.968837 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.968853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.970970 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.970975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.972700 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.972717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.975237 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.975245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.975441 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.976710 (+ 1269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.976739 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.981804 (+ 5065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.986105 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.986112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.986117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.987015 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.987523 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.987530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.987535 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.987707 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.988397 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.988401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.988572 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.988743 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.989054 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6616,"thread_start_us":6979,"threads_started":1}
I20260620 14:54:18.989449 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.936671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:42267 (local address 127.21.159.254:32891)
0620 14:54:18.966207 (+ 29536us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.966216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.966243 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:54:18.966313 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.966321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.966419 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.966730 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.969030 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.970810 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.974346 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.975008 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.975175 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.982087 (+ 6912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.982096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.982101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.982143 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.985927 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.987189 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.987195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.987201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.987338 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.987830 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.987835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.987839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.988182 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.988303 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.988856 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.989058 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29359,"thread_start_us":26088,"threads_started":1}
I20260620 14:54:18.990080 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.926140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.21.159.254:32891)
0620 14:54:18.962355 (+ 36215us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.962362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.962383 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:54:18.962444 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.962449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.962530 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.962662 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.964883 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.968797 (+ 3914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.975355 (+ 6558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.975914 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.976055 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.980431 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.980440 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.980446 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.980489 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.984800 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.986324 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.986329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.986333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.986473 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.987620 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.987625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.987628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.988029 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.988233 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.989019 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.989238 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35664,"spinlock_wait_cycles":1280,"thread_start_us":40429,"threads_started":1}
I20260620 14:54:18.990470 29578 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32891
I20260620 14:54:18.990624 29578 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.991113 29578 heartbeater.cc:511] Master 127.21.159.254:32891 requested a full tablet report, sending...
I20260620 14:54:18.992137 28320 ts_manager.cc:194] Re-registered known tserver with Master: 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335)
I20260620 14:54:18.993300 28320 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:42267
I20260620 14:54:18.994522 28320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:19.002465 28320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:19.038321 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.025251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.21.159.193:35161)
0620 14:54:19.028956 (+ 3705us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.028968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.029006 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:54:19.029144 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.029156 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.029529 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.029828 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:19.030630 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.032610 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.035634 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.037222 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.037403 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.037597 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.037821 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.038014 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3475,"thread_start_us":3328,"threads_started":1}
I20260620 14:54:19.038368 29606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.024589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:35161 (local address 127.0.0.1:38638)
0620 14:54:19.027209 (+ 2620us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:19.027232 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:19.027359 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:19.029831 (+ 2472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:19.029840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:19.029884 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:19.030377 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.030389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.032786 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.032791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.035488 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.035501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.037433 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.037439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.037615 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.037651 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:54:19.037758 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:54:19.037913 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2420,"mutex_wait_us":1,"thread_start_us":1424,"threads_started":1}
I20260620 14:54:19.039963 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.024573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.21.159.195:43335)
0620 14:54:19.030571 (+ 5998us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.030579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.030614 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:54:19.030985 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.030994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.031293 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.032029 (+ 736us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:19.032235 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.033749 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.036866 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.038578 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.038789 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.038984 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.039468 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.039631 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5834,"thread_start_us":5667,"threads_started":1}
I20260620 14:54:19.039943 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.024260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43335 (local address 127.0.0.1:54304)
0620 14:54:19.026347 (+ 2087us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:19.026374 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:19.026554 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:19.031515 (+ 4961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:19.031520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:19.031547 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:19.031860 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.031868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.033937 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.033949 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.036674 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.036689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.039022 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.039032 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.039257 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.039281 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:54:19.039411 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:54:19.039550 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1909,"mutex_wait_us":2,"thread_start_us":3381,"threads_started":1}
I20260620 14:54:19.043032 29223 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2095a8fa5d450d91bafd553b8d54ce (DEFAULT_TABLE table=test-table [id=d52bca936f914958a3784ac994fdc41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:19.043907 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.025638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.21.159.194:41545)
0620 14:54:19.028407 (+ 2769us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.028421 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.034998 (+ 6577us) server_negotiation.cc:407] Connection header received
0620 14:54:19.035203 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.035214 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.035504 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.035719 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:19.036911 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.038435 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.040960 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.042459 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.042599 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.042758 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.043455 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.043623 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2553,"thread_start_us":2405,"threads_started":1}
I20260620 14:54:19.044099 29512 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2095a8fa5d450d91bafd553b8d54ce (DEFAULT_TABLE table=test-table [id=d52bca936f914958a3784ac994fdc41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:19.044659 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.025004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:41545 (local address 127.0.0.1:37814)
0620 14:54:19.034840 (+ 9836us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:19.034862 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:19.035013 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:19.035686 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:19.035691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:19.035717 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:19.036672 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.036685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.038609 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.038614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.040765 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.040778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.042984 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.042991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.043205 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.043238 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:54:19.043908 (+ 670us) client_negotiation.cc:240] Negotiation successful
0620 14:54:19.044259 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9262,"spinlock_wait_cycles":942336,"thread_start_us":9074,"threads_started":1}
I20260620 14:54:19.044831 29223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2095a8fa5d450d91bafd553b8d54ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:19.045449 29512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2095a8fa5d450d91bafd553b8d54ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:19.050148 29371 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2095a8fa5d450d91bafd553b8d54ce (DEFAULT_TABLE table=test-table [id=d52bca936f914958a3784ac994fdc41c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:19.051795 29371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2095a8fa5d450d91bafd553b8d54ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:19.058768 29612 tablet_bootstrap.cc:492] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Bootstrap starting.
I20260620 14:54:19.063632 29612 tablet_bootstrap.cc:654] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:19.064033 29613 tablet_bootstrap.cc:492] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Bootstrap starting.
I20260620 14:54:19.065266 29612 log.cc:824] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:19.068889 29613 tablet_bootstrap.cc:654] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:19.069283 29615 tablet_bootstrap.cc:492] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Bootstrap starting.
I20260620 14:54:19.070715 29613 log.cc:824] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:19.073307 29612 tablet_bootstrap.cc:492] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: No bootstrap required, opened a new log
I20260620 14:54:19.073625 29615 tablet_bootstrap.cc:654] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:19.073652 29612 ts_tablet_manager.cc:1397] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260620 14:54:19.074764 29613 tablet_bootstrap.cc:492] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: No bootstrap required, opened a new log
I20260620 14:54:19.075006 29615 log.cc:824] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:19.075114 29613 ts_tablet_manager.cc:1397] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 14:54:19.078967 29615 tablet_bootstrap.cc:492] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: No bootstrap required, opened a new log
I20260620 14:54:19.079346 29615 ts_tablet_manager.cc:1397] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260620 14:54:19.084148 29613 raft_consensus.cc:348] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.084746 29613 raft_consensus.cc:374] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:19.084856 29613 raft_consensus.cc:729] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687f2d289d8d44a884e8c417758b1413, State: Initialized, Role: FOLLOWER
I20260620 14:54:19.084895 29612 raft_consensus.cc:348] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.085379 29612 raft_consensus.cc:374] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:19.085482 29612 raft_consensus.cc:729] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0613f1ae65d84469b92626f98ee51cc2, State: Initialized, Role: FOLLOWER
I20260620 14:54:19.086071 29613 consensus_queue.cc:260] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.086148 29612 consensus_queue.cc:260] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.092794 29612 ts_tablet_manager.cc:1428] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
I20260620 14:54:19.092447 29615 raft_consensus.cc:348] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.092978 29615 raft_consensus.cc:374] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:19.093067 29615 raft_consensus.cc:729] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb466bdeae9b45daa55de809056af60e, State: Initialized, Role: FOLLOWER
I20260620 14:54:19.093317 29613 ts_tablet_manager.cc:1428] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Time spent starting tablet: real 0.018s user 0.005s sys 0.012s
I20260620 14:54:19.093649 29578 heartbeater.cc:503] Master 127.21.159.254:32891 was elected leader, sending a full tablet report...
I20260620 14:54:19.093760 29615 consensus_queue.cc:260] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.094854 29621 raft_consensus.cc:482] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:19.095314 29437 heartbeater.cc:503] Master 127.21.159.254:32891 was elected leader, sending a full tablet report...
I20260620 14:54:19.096359 29621 raft_consensus.cc:504] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.098363 29621 leader_election.cc:288] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161), bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545)
I20260620 14:54:19.100344 29615 ts_tablet_manager.cc:1428] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Time spent starting tablet: real 0.021s user 0.011s sys 0.005s
W20260620 14:54:19.101100 29291 tablet.cc:2369] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:19.106323 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.098839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:48243 (local address 127.21.159.193:35161)
0620 14:54:19.099247 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.099253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.099270 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:54:19.099338 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.099343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.099423 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.099507 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:19.100072 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.101160 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.103855 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.105097 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.105246 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.105319 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.105931 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.106063 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271}
I20260620 14:54:19.106328 29593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.098714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:35161 (local address 127.21.159.195:48243)
0620 14:54:19.098942 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:19.098959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:19.099092 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:19.099667 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:19.099672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:19.099696 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:19.099887 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.099898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.101367 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.101373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.103655 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.103672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.105465 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.105474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.105747 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.105773 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:54:19.105903 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:54:19.106053 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:54:19.108196 29243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2095a8fa5d450d91bafd553b8d54ce" candidate_uuid: "687f2d289d8d44a884e8c417758b1413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0613f1ae65d84469b92626f98ee51cc2" is_pre_election: true
I20260620 14:54:19.108801 29243 raft_consensus.cc:2454] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687f2d289d8d44a884e8c417758b1413 in term 0.
I20260620 14:54:19.109699 29467 leader_election.cc:302] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0613f1ae65d84469b92626f98ee51cc2, 687f2d289d8d44a884e8c417758b1413; no voters:
I20260620 14:54:19.109859 29582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.100643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:41545 (local address 127.21.159.195:56011)
0620 14:54:19.101165 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:19.101184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:19.101277 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:19.103950 (+ 2673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:19.103954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:19.103979 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:19.104165 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.104175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.105494 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.105499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.107423 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:19.107437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.109034 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.109041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:19.109217 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.109260 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:54:19.109360 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:54:19.109540 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382}
I20260620 14:54:19.110394 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.103361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:56011 (local address 127.21.159.194:41545)
0620 14:54:19.103540 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.103546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.103565 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:54:19.103623 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.103628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.103692 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.103807 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:19.104318 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.105353 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.107599 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.108847 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.109006 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.109071 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.109981 (+ 910us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.110117 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:54:19.110459 29621 raft_consensus.cc:2781] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:19.110711 29621 raft_consensus.cc:482] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:19.110827 29621 raft_consensus.cc:3037] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:19.112484 29391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2095a8fa5d450d91bafd553b8d54ce" candidate_uuid: "687f2d289d8d44a884e8c417758b1413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb466bdeae9b45daa55de809056af60e" is_pre_election: true
I20260620 14:54:19.113051 29391 raft_consensus.cc:2454] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687f2d289d8d44a884e8c417758b1413 in term 0.
I20260620 14:54:19.114290 29621 raft_consensus.cc:504] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.115518 29391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2095a8fa5d450d91bafd553b8d54ce" candidate_uuid: "687f2d289d8d44a884e8c417758b1413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb466bdeae9b45daa55de809056af60e"
I20260620 14:54:19.115694 29243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2095a8fa5d450d91bafd553b8d54ce" candidate_uuid: "687f2d289d8d44a884e8c417758b1413" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0613f1ae65d84469b92626f98ee51cc2"
I20260620 14:54:19.115728 29391 raft_consensus.cc:3037] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:19.115899 29243 raft_consensus.cc:3037] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:19.118052 29621 leader_election.cc:288] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [CANDIDATE]: Term 1 election: Requested vote from peers 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161), bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545)
I20260620 14:54:19.119019 29243 raft_consensus.cc:2454] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687f2d289d8d44a884e8c417758b1413 in term 1.
I20260620 14:54:19.119246 29391 raft_consensus.cc:2454] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687f2d289d8d44a884e8c417758b1413 in term 1.
I20260620 14:54:19.119491 29467 leader_election.cc:302] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0613f1ae65d84469b92626f98ee51cc2, 687f2d289d8d44a884e8c417758b1413; no voters:
I20260620 14:54:19.119803 29621 raft_consensus.cc:2781] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:19.119989 29621 raft_consensus.cc:686] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 1 LEADER]: Becoming Leader. State: Replica: 687f2d289d8d44a884e8c417758b1413, State: Running, Role: LEADER
I20260620 14:54:19.120463 29621 consensus_queue.cc:237] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } }
I20260620 14:54:19.125560 28320 catalog_manager.cc:5495] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 reported cstate change: term changed from 0 to 1, leader changed from <none> to 687f2d289d8d44a884e8c417758b1413 (127.21.159.195). New cstate: current_term: 1 leader_uuid: "687f2d289d8d44a884e8c417758b1413" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:19.150758 29608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.144872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.21.159.195:43335)
0620 14:54:19.145115 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.145122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.145238 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:54:19.145422 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.145426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.145502 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.145669 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:19.146040 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.146810 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.148675 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.149097 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.149196 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.149634 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:19.150203 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:19.150327 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.150488 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.150557 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:54:19.159464 29243 raft_consensus.cc:1261] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Refusing update from remote peer 687f2d289d8d44a884e8c417758b1413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:19.159449 29391 raft_consensus.cc:1261] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 1 FOLLOWER]: Refusing update from remote peer 687f2d289d8d44a884e8c417758b1413: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:19.160544 29621 consensus_queue.cc:1035] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:19.160881 29619 consensus_queue.cc:1035] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:54:19.172379 29580 tablet.cc:2369] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:19.192602 28320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:19.192893 28320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:19.196558 29626 mvcc.cc:205] Tried to move back new op lower bound from 7298937893505748992 to 7298937893360857088. Current Snapshot: MvccSnapshot[applied={T|T < 7298937893505748992}]
I20260620 14:54:19.205156 28320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c179d685f7f4ba68e8a0760536cbb73: Sending DeleteTablet for 3 replicas of tablet 8e2095a8fa5d450d91bafd553b8d54ce
I20260620 14:54:19.206496 29371 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2095a8fa5d450d91bafd553b8d54ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:19 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:37814
I20260620 14:54:19.206971 29512 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2095a8fa5d450d91bafd553b8d54ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:19 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54304
I20260620 14:54:19.208309 29223 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2095a8fa5d450d91bafd553b8d54ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:19 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:38638
I20260620 14:54:19.208886 29222 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e2095a8fa5d450d91bafd553b8d54ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:19 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:38638
I20260620 14:54:19.211894 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 28288
I20260620 14:54:19.213315 28350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:19.231643 29635 tablet_replica.cc:330] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: stopping tablet replica
I20260620 14:54:19.232147 29635 raft_consensus.cc:2229] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:19.232671 29635 raft_consensus.cc:2258] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:19.233600 29634 tablet_replica.cc:330] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: stopping tablet replica
I20260620 14:54:19.234047 29634 raft_consensus.cc:2229] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:19.234722 29634 raft_consensus.cc:2258] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:19.237155 29635 ts_tablet_manager.cc:1905] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:19.237219 29634 ts_tablet_manager.cc:1905] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:19.244772 29635 ts_tablet_manager.cc:1918] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:19.244958 29635 log.cc:1196] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/8e2095a8fa5d450d91bafd553b8d54ce
I20260620 14:54:19.245548 29635 ts_tablet_manager.cc:1939] T 8e2095a8fa5d450d91bafd553b8d54ce P 0613f1ae65d84469b92626f98ee51cc2: Deleting consensus metadata
I20260620 14:54:19.246785 29634 ts_tablet_manager.cc:1918] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:19.246980 29634 log.cc:1196] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/8e2095a8fa5d450d91bafd553b8d54ce
I20260620 14:54:19.247257 29633 tablet_replica.cc:330] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: stopping tablet replica
I20260620 14:54:19.247637 29634 ts_tablet_manager.cc:1939] T 8e2095a8fa5d450d91bafd553b8d54ce P 687f2d289d8d44a884e8c417758b1413: Deleting consensus metadata
I20260620 14:54:19.247746 29633 raft_consensus.cc:2229] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:19.248349 29633 raft_consensus.cc:2258] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:19.250975 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.209437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.21.159.253:35607)
0620 14:54:19.209606 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.209615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.210584 (+ 969us) server_negotiation.cc:407] Connection header received
0620 14:54:19.210876 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.210884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.210981 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.211082 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:19.216062 (+ 4980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.217491 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.220549 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.221413 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.221557 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.227128 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:19.227139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:19.227147 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:19.227213 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:19.236601 (+ 9388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.246664 (+ 10063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.246671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.246675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.247410 (+ 735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.248400 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.248406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.248412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.248911 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:19.249153 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.249852 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.250161 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:54:19.251235 29633 ts_tablet_manager.cc:1905] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:19.259958 29633 ts_tablet_manager.cc:1918] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:19.260144 29633 log.cc:1196] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/8e2095a8fa5d450d91bafd553b8d54ce
I20260620 14:54:19.260703 29633 ts_tablet_manager.cc:1939] T 8e2095a8fa5d450d91bafd553b8d54ce P bb466bdeae9b45daa55de809056af60e: Deleting consensus metadata
I20260620 14:54:19.278702 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.211580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.21.159.252:39419)
0620 14:54:19.211739 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.211745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.253908 (+ 42163us) server_negotiation.cc:407] Connection header received
0620 14:54:19.254154 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.254161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.254239 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.254301 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:19.259210 (+ 4909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.260366 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.263259 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.263941 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.264208 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.268956 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:19.268964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:19.268969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:19.271211 (+ 2242us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:19.275132 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.276103 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.276110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.276114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.276259 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.276673 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.276678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.276682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.277043 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:19.277302 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.277919 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.278178 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:54:19.402127 28304 catalog_manager.cc:4841] TS bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545): tablet 8e2095a8fa5d450d91bafd553b8d54ce (table test-table [id=d52bca936f914958a3784ac994fdc41c]) successfully deleted
I20260620 14:54:19.403035 28306 catalog_manager.cc:4841] TS 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161): tablet 8e2095a8fa5d450d91bafd553b8d54ce (table test-table [id=d52bca936f914958a3784ac994fdc41c]) successfully deleted
I20260620 14:54:19.403242 28305 catalog_manager.cc:4841] TS 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335): tablet 8e2095a8fa5d450d91bafd553b8d54ce (table test-table [id=d52bca936f914958a3784ac994fdc41c]) successfully deleted
W20260620 14:54:19.403580 28306 catalog_manager.cc:4808] TS 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161): delete failed for tablet 8e2095a8fa5d450d91bafd553b8d54ce because the tablet was not found. No further retry: Not found: Tablet not found: 8e2095a8fa5d450d91bafd553b8d54ce
I20260620 14:54:19.403821 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 28288
I20260620 14:54:19.456257 29583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.431880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.21.159.253:35607)
0620 14:54:19.432170 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.432180 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.433370 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 14:54:19.433696 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.433707 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.433829 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.433931 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:19.437506 (+ 3575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.439432 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.442582 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.443736 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.443895 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.449221 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:19.449229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:19.449234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:19.449289 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:19.453038 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.453888 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.453895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.453899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.454100 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.454610 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.454614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.454616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.454943 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:19.455282 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.455668 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.455965 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:54:19.463682 29581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:19.433445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58198 (local address 127.21.159.252:39419)
0620 14:54:19.433677 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:19.433683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:19.444643 (+ 10960us) server_negotiation.cc:407] Connection header received
0620 14:54:19.445347 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:19.445356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:19.445455 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:19.445541 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:19.449212 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.450389 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.452329 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:19.453062 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:19.453229 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:19.457304 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:19.457312 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:19.457316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:19.457372 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:19.460383 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.461347 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.461351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.461354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.461489 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:19.462053 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:19.462058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:19.462060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:19.462430 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:19.462630 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:19.463019 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 14:54:19.463277 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:54:20.195991 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.194979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.195:46259)
0620 14:54:20.195568 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.195795 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":348,"threads_started":1}
W20260620 14:54:20.196378 29578 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:32891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
I20260620 14:54:20.209465 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.208709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.193:38603)
0620 14:54:20.209187 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.209295 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":266,"threads_started":1}
W20260620 14:54:20.209901 29289 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:32891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
I20260620 14:54:20.211817 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.211156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:55637)
0620 14:54:20.211538 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.211644 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":192,"threads_started":1}
W20260620 14:54:20.212221 29437 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:32891 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
I20260620 14:54:20.705173 29648 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c179d685f7f4ba68e8a0760536cbb73)
I20260620 14:54:20.705562 29648 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } }
I20260620 14:54:20.706066 29649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5c179d685f7f4ba68e8a0760536cbb73)
I20260620 14:54:20.706323 29649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } }
I20260620 14:54:20.707459 29648 leader_election.cc:288] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891), 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252:39419)
I20260620 14:54:20.707566 29649 leader_election.cc:288] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891), dc4a2c20c0a4469da2830f9ad5355817 (127.21.159.253:35607)
I20260620 14:54:20.708918 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.707708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38088)
0620 14:54:20.708373 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.708560 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":314,"threads_started":1}
I20260620 14:54:20.710841 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.707712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38092)
0620 14:54:20.710171 (+ 2459us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.710444 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2257,"threads_started":1}
W20260620 14:54:20.712354 28906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
W20260620 14:54:20.713327 29094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
I20260620 14:54:20.716887 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.716071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38096)
0620 14:54:20.716441 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.716658 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260620 14:54:20.717065 29094 leader_election.cc:334] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891): Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
I20260620 14:54:20.717962 29654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.717351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38108)
0620 14:54:20.717551 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.717775 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:54:20.718258 28906 leader_election.cc:334] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891): Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
I20260620 14:54:20.719908 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.710096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.21.159.252:39419)
0620 14:54:20.711933 (+ 1837us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.711939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.711959 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:54:20.712021 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.712026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.712108 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.712259 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:20.713103 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.714278 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.717682 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.718887 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.719005 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.719121 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.719472 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.719580 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1707,"thread_start_us":1553,"threads_started":1}
I20260620 14:54:20.719935 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.708463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:39419 (local address 127.0.0.1:58206)
0620 14:54:20.709221 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.709248 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:20.709433 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:20.712290 (+ 2857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:20.712295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:20.712346 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:20.712901 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.712920 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.714550 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.714558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.717491 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.717506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.719079 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.719086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.719288 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.719326 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:54:20.719435 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:54:20.719605 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":3406,"threads_started":1}
I20260620 14:54:20.720624 29120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc4a2c20c0a4469da2830f9ad5355817" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" is_pre_election: true
I20260620 14:54:20.721012 29120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc4a2c20c0a4469da2830f9ad5355817 in term 1.
I20260620 14:54:20.721621 28906 leader_election.cc:302] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ff9d3e39cc647e5b400c285b5df56f2, dc4a2c20c0a4469da2830f9ad5355817; no voters: 5c179d685f7f4ba68e8a0760536cbb73
I20260620 14:54:20.722077 29648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:54:20.722154 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.707888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:35607 (local address 127.0.0.1:53312)
0620 14:54:20.711150 (+ 3262us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.711177 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:20.711360 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:20.714292 (+ 2932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:20.714298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:20.714333 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:20.714774 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.714787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.717520 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.717531 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.719717 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.719733 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.721339 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.721345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.721521 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.721544 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:54:20.721656 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:54:20.721804 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3090,"mutex_wait_us":87,"thread_start_us":1874,"threads_started":1}
I20260620 14:54:20.722250 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.708008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.21.159.253:35607)
0620 14:54:20.713816 (+ 5808us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.713825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.713846 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:54:20.713904 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.713912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.714009 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.715185 (+ 1176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:20.715574 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.717348 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.719893 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.721148 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.721339 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.721411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.721688 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.721815 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5648,"thread_start_us":5794,"threads_started":1}
I20260620 14:54:20.722224 29648 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5c179d685f7f4ba68e8a0760536cbb73)
I20260620 14:54:20.722376 29648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:54:20.722756 28931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dc4a2c20c0a4469da2830f9ad5355817" is_pre_election: true
I20260620 14:54:20.724905 29648 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } }
I20260620 14:54:20.725423 28931 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ff9d3e39cc647e5b400c285b5df56f2 in current term 2: Already voted for candidate dc4a2c20c0a4469da2830f9ad5355817 in this term.
I20260620 14:54:20.725713 29648 leader_election.cc:288] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [CANDIDATE]: Term 2 election: Requested vote from peers 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891), 9ff9d3e39cc647e5b400c285b5df56f2 (127.21.159.252:39419)
I20260620 14:54:20.726279 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.725840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38116)
0620 14:54:20.725983 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.726104 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:54:20.726589 29097 leader_election.cc:302] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ff9d3e39cc647e5b400c285b5df56f2; no voters: 5c179d685f7f4ba68e8a0760536cbb73, dc4a2c20c0a4469da2830f9ad5355817
I20260620 14:54:20.726904 29649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:54:20.727108 29120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc4a2c20c0a4469da2830f9ad5355817" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9ff9d3e39cc647e5b400c285b5df56f2"
I20260620 14:54:20.727347 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.726932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38132)
0620 14:54:20.727125 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.727224 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:54:20.727543 28906 leader_election.cc:334] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891): Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
I20260620 14:54:20.729478 29649 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:54:20.731855 29120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc4a2c20c0a4469da2830f9ad5355817 in term 2.
I20260620 14:54:20.732642 28906 leader_election.cc:302] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ff9d3e39cc647e5b400c285b5df56f2, dc4a2c20c0a4469da2830f9ad5355817; no voters: 5c179d685f7f4ba68e8a0760536cbb73
I20260620 14:54:20.733016 29648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:54:20.733563 29648 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [term 2 LEADER]: Becoming Leader. State: Replica: dc4a2c20c0a4469da2830f9ad5355817, State: Running, Role: LEADER
I20260620 14:54:20.734133 29648 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } }
I20260620 14:54:20.736393 29658 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc4a2c20c0a4469da2830f9ad5355817. Latest consensus state: current_term: 2 leader_uuid: "dc4a2c20c0a4469da2830f9ad5355817" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:20.736873 29658 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:20.737679 29660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:20.741676 29660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:20.742676 29660 catalog_manager.cc:1259] Loaded cluster ID: 4ef396ced08041b484a5914dc262644d
I20260620 14:54:20.742758 29660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:20.744262 29660 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:20.745265 29660 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Loaded TSK: 0
I20260620 14:54:20.746292 29660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:20.763535 28922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:20.770938 28922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:20.775631 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.775159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38140)
0620 14:54:20.775396 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.775486 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:54:20.776023 29120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [term 2 FOLLOWER]: Refusing update from remote peer dc4a2c20c0a4469da2830f9ad5355817: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 14:54:20.776798 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.776382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38156)
0620 14:54:20.776548 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.776636 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:54:20.776922 29658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 14:54:20.777267 28906 consensus_peers.cc:487] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 -> Peer 5c179d685f7f4ba68e8a0760536cbb73 (127.21.159.254:32891): Couldn't send request to peer 5c179d685f7f4ba68e8a0760536cbb73. Status: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:54:20.780045 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc4a2c20c0a4469da2830f9ad5355817. Latest consensus state: current_term: 2 leader_uuid: "dc4a2c20c0a4469da2830f9ad5355817" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:20.780426 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:54:20.781991 29648 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "dc4a2c20c0a4469da2830f9ad5355817" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:20.782342 29648 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:20.782701 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "dc4a2c20c0a4469da2830f9ad5355817" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5c179d685f7f4ba68e8a0760536cbb73" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32891 } } peers { permanent_uuid: "dc4a2c20c0a4469da2830f9ad5355817" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 35607 } attrs { promote: false } } peers { permanent_uuid: "9ff9d3e39cc647e5b400c285b5df56f2" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 39419 } attrs { promote: false } } }
I20260620 14:54:20.783082 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ff9d3e39cc647e5b400c285b5df56f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:54:20.804032 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.792751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.21.159.195:43335)
0620 14:54:20.793236 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.793241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.794513 (+ 1272us) server_negotiation.cc:407] Connection header received
0620 14:54:20.794621 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.794629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.794737 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.794870 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:20.795665 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.797177 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.800807 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.802305 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.802500 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.802605 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.803482 (+ 877us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.803673 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260620 14:54:20.804149 29668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.792682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43335 (local address 127.0.0.1:54324)
0620 14:54:20.793274 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.793302 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:20.793441 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:20.794906 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:20.794913 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:20.794956 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:20.795456 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.795472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.797463 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.797472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.800540 (+ 3068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.800566 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.802918 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.802924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.803208 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.803325 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 14:54:20.803431 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:54:20.803681 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":131,"thread_start_us":478,"threads_started":1}
I20260620 14:54:20.805388 29512 tablet_service.cc:1467] Processing CreateTablet for tablet a6dfd6b72d14484ca9bae38b1e9ccb23 (DEFAULT_TABLE table=test-table [id=9d1cd03fd211463a9f57fbb0fc5f1c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:20.805902 29512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dfd6b72d14484ca9bae38b1e9ccb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:20.807261 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.792148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:41545 (local address 127.0.0.1:37822)
0620 14:54:20.792369 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.792395 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:20.792534 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:20.801422 (+ 8888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:20.801427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:20.801454 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:20.801651 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.801664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.803222 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.803226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.805141 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.805152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.806385 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.806391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.806642 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.806678 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:54:20.806771 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:54:20.806931 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":47}
I20260620 14:54:20.808621 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.792244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.21.159.194:41545)
0620 14:54:20.801123 (+ 8879us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.801128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.801144 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:54:20.801202 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.801205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.801264 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.801401 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:20.801841 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.803039 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.805306 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.806213 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.806387 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.806468 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.807183 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.808321 (+ 1138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8767,"thread_start_us":8664,"threads_started":1}
I20260620 14:54:20.808889 29654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.792246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:35161 (local address 127.0.0.1:38640)
0620 14:54:20.792632 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.792657 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:20.792756 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:20.802304 (+ 9548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:20.802311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:20.802339 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:20.802702 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.802713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.804667 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.804671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.806512 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.806527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.808118 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.808124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.808308 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.808332 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:54:20.808445 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:54:20.808585 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":1}
I20260620 14:54:20.808892 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.793035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.21.159.193:35161)
0620 14:54:20.801900 (+ 8865us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.801908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.801930 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:54:20.802009 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.802014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.802097 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.803313 (+ 1216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:20.803485 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.804529 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.806693 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.807911 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.808103 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.808172 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.808473 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.808612 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8714,"thread_start_us":8617,"threads_started":1}
I20260620 14:54:20.809800 29371 tablet_service.cc:1467] Processing CreateTablet for tablet a6dfd6b72d14484ca9bae38b1e9ccb23 (DEFAULT_TABLE table=test-table [id=9d1cd03fd211463a9f57fbb0fc5f1c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:20.810223 29371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dfd6b72d14484ca9bae38b1e9ccb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:20.812071 29223 tablet_service.cc:1467] Processing CreateTablet for tablet a6dfd6b72d14484ca9bae38b1e9ccb23 (DEFAULT_TABLE table=test-table [id=9d1cd03fd211463a9f57fbb0fc5f1c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:20.812330 29671 tablet_bootstrap.cc:492] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: Bootstrap starting.
I20260620 14:54:20.812511 29223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6dfd6b72d14484ca9bae38b1e9ccb23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:20.813889 29671 tablet_bootstrap.cc:654] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:20.816352 29671 tablet_bootstrap.cc:492] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: No bootstrap required, opened a new log
I20260620 14:54:20.816538 29671 ts_tablet_manager.cc:1397] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:54:20.817258 29671 raft_consensus.cc:348] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.817483 29671 raft_consensus.cc:374] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:20.817569 29671 raft_consensus.cc:729] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687f2d289d8d44a884e8c417758b1413, State: Initialized, Role: FOLLOWER
I20260620 14:54:20.817828 29671 consensus_queue.cc:260] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.819975 29671 ts_tablet_manager.cc:1428] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:20.822530 29579 heartbeater.cc:503] Master 127.21.159.253:35607 was elected leader, sending a full tablet report...
I20260620 14:54:20.825111 29676 tablet_bootstrap.cc:492] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: Bootstrap starting.
I20260620 14:54:20.826064 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.819005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.195:35929)
0620 14:54:20.825580 (+ 6575us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.825767 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6446,"thread_start_us":6323,"threads_started":1}
I20260620 14:54:20.827756 29675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.827047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.195:41759)
0620 14:54:20.827316 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.827498 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"threads_started":1}
I20260620 14:54:20.828639 29676 tablet_bootstrap.cc:654] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:20.829499 29673 tablet_bootstrap.cc:492] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: Bootstrap starting.
I20260620 14:54:20.831228 29673 tablet_bootstrap.cc:654] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:20.833552 29676 tablet_bootstrap.cc:492] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: No bootstrap required, opened a new log
I20260620 14:54:20.833652 29673 tablet_bootstrap.cc:492] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: No bootstrap required, opened a new log
I20260620 14:54:20.833804 29673 ts_tablet_manager.cc:1397] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:54:20.833875 29676 ts_tablet_manager.cc:1397] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 14:54:20.834506 29673 raft_consensus.cc:348] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.834565 29676 raft_consensus.cc:348] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.834745 29673 raft_consensus.cc:374] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:20.834733 29676 raft_consensus.cc:374] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:20.834879 29676 raft_consensus.cc:729] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0613f1ae65d84469b92626f98ee51cc2, State: Initialized, Role: FOLLOWER
I20260620 14:54:20.834888 29673 raft_consensus.cc:729] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb466bdeae9b45daa55de809056af60e, State: Initialized, Role: FOLLOWER
I20260620 14:54:20.835193 29673 consensus_queue.cc:260] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.835202 29676 consensus_queue.cc:260] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.836040 29673 ts_tablet_manager.cc:1428] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:54:20.840160 29676 ts_tablet_manager.cc:1428] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 14:54:20.840459 29290 heartbeater.cc:503] Master 127.21.159.253:35607 was elected leader, sending a full tablet report...
I20260620 14:54:20.841603 29438 heartbeater.cc:503] Master 127.21.159.253:35607 was elected leader, sending a full tablet report...
I20260620 14:54:20.844050 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.836626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:57953)
0620 14:54:20.843544 (+ 6918us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.843748 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4483,"thread_start_us":4318,"threads_started":1}
I20260620 14:54:20.844061 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.836624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.193:53845)
0620 14:54:20.843527 (+ 6903us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.843735 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6761,"thread_start_us":6632,"threads_started":1}
I20260620 14:54:20.844993 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.844641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:32891)
0620 14:54:20.844801 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.844879 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:54:20.845585 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.845195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.193:60667)
0620 14:54:20.845344 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.845446 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:54:20.872890 29680 raft_consensus.cc:482] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:20.873127 29680 raft_consensus.cc:504] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.874648 29680 leader_election.cc:288] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161), 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335)
I20260620 14:54:20.882010 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.875111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:38671 (local address 127.21.159.195:43335)
0620 14:54:20.875320 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.875367 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.875706 (+ 339us) server_negotiation.cc:407] Connection header received
0620 14:54:20.876002 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.876008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.876094 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.876208 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:20.876833 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.877666 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.879993 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.881046 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.881209 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.881289 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.881592 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.881726 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:54:20.882091 29683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.875112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43335 (local address 127.21.159.194:38671)
0620 14:54:20.875562 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.875588 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:20.875816 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:20.876260 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:20.876265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:20.876319 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:20.876681 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.876693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.877801 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.877806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.879814 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.879829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.881214 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.881221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.881426 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.881450 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:54:20.881567 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:54:20.881712 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":74,"spinlock_wait_cycles":1920}
I20260620 14:54:20.882135 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.875127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:43205 (local address 127.21.159.193:35161)
0620 14:54:20.875333 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.875339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.876278 (+ 939us) server_negotiation.cc:407] Connection header received
0620 14:54:20.876462 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.876470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.876546 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.876656 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:20.877226 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.878118 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.880134 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.881166 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.881357 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.881430 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.881745 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.881873 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:54:20.882652 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.875144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:35161 (local address 127.21.159.194:43205)
0620 14:54:20.876152 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.876178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:20.876306 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:20.876673 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:20.876675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:20.876693 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:20.877063 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.877076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.878250 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.878253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.879957 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:20.879971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.881384 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.881390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:20.881581 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.881606 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:54:20.881707 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:54:20.881840 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":2,"spinlock_wait_cycles":3968,"thread_start_us":703,"threads_started":1}
I20260620 14:54:20.883023 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dfd6b72d14484ca9bae38b1e9ccb23" candidate_uuid: "bb466bdeae9b45daa55de809056af60e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687f2d289d8d44a884e8c417758b1413" is_pre_election: true
I20260620 14:54:20.883412 29242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dfd6b72d14484ca9bae38b1e9ccb23" candidate_uuid: "bb466bdeae9b45daa55de809056af60e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0613f1ae65d84469b92626f98ee51cc2" is_pre_election: true
I20260620 14:54:20.883595 29532 raft_consensus.cc:2454] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb466bdeae9b45daa55de809056af60e in term 0.
I20260620 14:54:20.883706 29242 raft_consensus.cc:2454] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb466bdeae9b45daa55de809056af60e in term 0.
I20260620 14:54:20.884239 29325 leader_election.cc:302] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687f2d289d8d44a884e8c417758b1413, bb466bdeae9b45daa55de809056af60e; no voters:
I20260620 14:54:20.884672 29680 raft_consensus.cc:2781] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:20.884809 29680 raft_consensus.cc:482] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:20.884920 29680 raft_consensus.cc:3037] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:20.887040 29680 raft_consensus.cc:504] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.887719 29680 leader_election.cc:288] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [CANDIDATE]: Term 1 election: Requested vote from peers 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161), 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335)
I20260620 14:54:20.888274 29532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dfd6b72d14484ca9bae38b1e9ccb23" candidate_uuid: "bb466bdeae9b45daa55de809056af60e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "687f2d289d8d44a884e8c417758b1413"
I20260620 14:54:20.888274 29242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6dfd6b72d14484ca9bae38b1e9ccb23" candidate_uuid: "bb466bdeae9b45daa55de809056af60e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0613f1ae65d84469b92626f98ee51cc2"
I20260620 14:54:20.888489 29532 raft_consensus.cc:3037] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:20.888509 29242 raft_consensus.cc:3037] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:20.891546 29242 raft_consensus.cc:2454] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb466bdeae9b45daa55de809056af60e in term 1.
I20260620 14:54:20.891554 29532 raft_consensus.cc:2454] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb466bdeae9b45daa55de809056af60e in term 1.
I20260620 14:54:20.892059 29326 leader_election.cc:302] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0613f1ae65d84469b92626f98ee51cc2, 687f2d289d8d44a884e8c417758b1413, bb466bdeae9b45daa55de809056af60e; no voters:
I20260620 14:54:20.892473 29680 raft_consensus.cc:2781] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:20.892947 29680 raft_consensus.cc:686] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 1 LEADER]: Becoming Leader. State: Replica: bb466bdeae9b45daa55de809056af60e, State: Running, Role: LEADER
I20260620 14:54:20.893440 29680 consensus_queue.cc:237] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } }
I20260620 14:54:20.894249 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.893840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:47671)
0620 14:54:20.894004 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.894114 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:54:20.897167 28921 catalog_manager.cc:5495] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb466bdeae9b45daa55de809056af60e (127.21.159.194). New cstate: current_term: 1 leader_uuid: "bb466bdeae9b45daa55de809056af60e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb466bdeae9b45daa55de809056af60e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 41545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:20.937441 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.931675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37828 (local address 127.21.159.194:41545)
0620 14:54:20.931908 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.931915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.931987 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:54:20.932267 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.932272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.932394 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.932560 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:20.932958 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.933652 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.935028 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:20.935532 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:20.935678 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:20.936143 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:20.936804 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:20.936965 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:20.937074 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:54:20.937161 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:54:20.941962 29242 raft_consensus.cc:1261] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Refusing update from remote peer bb466bdeae9b45daa55de809056af60e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:20.942427 29532 raft_consensus.cc:1261] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 1 FOLLOWER]: Refusing update from remote peer bb466bdeae9b45daa55de809056af60e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:20.942741 29685 consensus_queue.cc:1035] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0613f1ae65d84469b92626f98ee51cc2" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 35161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:20.943225 29680 consensus_queue.cc:1035] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [LEADER]: Connected to new peer: Peer: permanent_uuid: "687f2d289d8d44a884e8c417758b1413" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:20.946375 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.945399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.195:50539)
0620 14:54:20.945768 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.946042 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260620 14:54:20.946924 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.945492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.193:40089)
0620 14:54:20.946640 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.946763 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1035}
I20260620 14:54:20.949074 29688 mvcc.cc:205] Tried to move back new op lower bound from 7298937900809986048 to 7298937900623831040. Current Snapshot: MvccSnapshot[applied={T|T < 7298937900809986048}]
I20260620 14:54:20.951894 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.951223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.195:52979)
0620 14:54:20.951524 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.951673 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 14:54:20.954596 29690 mvcc.cc:205] Tried to move back new op lower bound from 7298937900809986048 to 7298937900623831040. Current Snapshot: MvccSnapshot[applied={T|T < 7298937900809986048 or (T in {7298937900809986048})}]
I20260620 14:54:20.958072 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.957525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:44917)
0620 14:54:20.957747 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.957883 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 14:54:20.962386 29682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.961923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.193:44225)
0620 14:54:20.962107 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.962206 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:54:20.963234 29684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.962785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.21.159.194:54739)
0620 14:54:20.962941 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:20.963082 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:54:20.975275 28921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:20.975572 28921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:20.981483 28921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc4a2c20c0a4469da2830f9ad5355817: Sending DeleteTablet for 3 replicas of tablet a6dfd6b72d14484ca9bae38b1e9ccb23
I20260620 14:54:20.982636 29371 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dfd6b72d14484ca9bae38b1e9ccb23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:20 UTC) from {username='oryx', principal='oryx/127.21.159.253@KRBTEST.COM'} at 127.0.0.1:37822
I20260620 14:54:20.982998 29512 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dfd6b72d14484ca9bae38b1e9ccb23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:20 UTC) from {username='oryx', principal='oryx/127.21.159.253@KRBTEST.COM'} at 127.0.0.1:54324
I20260620 14:54:20.983668 29223 tablet_service.cc:1514] Processing DeleteTablet for tablet a6dfd6b72d14484ca9bae38b1e9ccb23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:20 UTC) from {username='oryx', principal='oryx/127.21.159.253@KRBTEST.COM'} at 127.0.0.1:38640
I20260620 14:54:20.985921 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 29093
I20260620 14:54:20.994006 29700 tablet_replica.cc:330] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: stopping tablet replica
I20260620 14:54:20.994505 29700 raft_consensus.cc:2229] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:20.995038 29700 raft_consensus.cc:2258] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:20.996833 29700 ts_tablet_manager.cc:1905] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:20.999531 29699 tablet_replica.cc:330] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: stopping tablet replica
I20260620 14:54:20.999827 29699 raft_consensus.cc:2229] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:21.000234 29699 raft_consensus.cc:2258] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:21.001592 29699 ts_tablet_manager.cc:1905] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:21.005743 29700 ts_tablet_manager.cc:1918] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:21.005997 29700 log.cc:1196] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/a6dfd6b72d14484ca9bae38b1e9ccb23
I20260620 14:54:21.006536 29700 ts_tablet_manager.cc:1939] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 0613f1ae65d84469b92626f98ee51cc2: Deleting consensus metadata
I20260620 14:54:21.008054 28908 catalog_manager.cc:4841] TS 0613f1ae65d84469b92626f98ee51cc2 (127.21.159.193:35161): tablet a6dfd6b72d14484ca9bae38b1e9ccb23 (table test-table [id=9d1cd03fd211463a9f57fbb0fc5f1c72]) successfully deleted
I20260620 14:54:21.009404 29698 tablet_replica.cc:330] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: stopping tablet replica
I20260620 14:54:21.009660 29698 raft_consensus.cc:2229] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:21.010397 29698 raft_consensus.cc:2258] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:21.012028 29698 ts_tablet_manager.cc:1905] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:21.012377 29699 ts_tablet_manager.cc:1918] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:21.012527 29699 log.cc:1196] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/a6dfd6b72d14484ca9bae38b1e9ccb23
I20260620 14:54:21.013023 29699 ts_tablet_manager.cc:1939] T a6dfd6b72d14484ca9bae38b1e9ccb23 P 687f2d289d8d44a884e8c417758b1413: Deleting consensus metadata
I20260620 14:54:21.014338 28907 catalog_manager.cc:4841] TS 687f2d289d8d44a884e8c417758b1413 (127.21.159.195:43335): tablet a6dfd6b72d14484ca9bae38b1e9ccb23 (table test-table [id=9d1cd03fd211463a9f57fbb0fc5f1c72]) successfully deleted
I20260620 14:54:21.019814 29698 ts_tablet_manager.cc:1918] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:21.019985 29698 log.cc:1196] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/a6dfd6b72d14484ca9bae38b1e9ccb23
I20260620 14:54:21.020527 29698 ts_tablet_manager.cc:1939] T a6dfd6b72d14484ca9bae38b1e9ccb23 P bb466bdeae9b45daa55de809056af60e: Deleting consensus metadata
I20260620 14:54:21.021763 28906 catalog_manager.cc:4841] TS bb466bdeae9b45daa55de809056af60e (127.21.159.194:41545): tablet a6dfd6b72d14484ca9bae38b1e9ccb23 (table test-table [id=9d1cd03fd211463a9f57fbb0fc5f1c72]) successfully deleted
I20260620 14:54:21.023969 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:20.986821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.21.159.193:35161)
0620 14:54:20.987069 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:20.987080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:20.999399 (+ 12319us) server_negotiation.cc:407] Connection header received
0620 14:54:20.999790 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:20.999796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:20.999905 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:20.999988 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:21.008898 (+ 8910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:21.010033 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:21.012070 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:21.012748 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:21.012873 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:21.017066 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:21.017082 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:21.017098 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:21.017139 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:21.020427 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:21.021286 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:21.021295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:21.021298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:21.021495 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:21.022010 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:21.022014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:21.022017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:21.022464 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:21.022682 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:21.023308 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 14:54:21.023578 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:54:21.024845 29283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:21.124925 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 29093
I20260620 14:54:21.143235 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 29300
I20260620 14:54:21.163997 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:21.144058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.21.159.194:41545)
0620 14:54:21.144247 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:21.144254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:21.144276 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:54:21.144444 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:21.144450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:21.144575 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:21.144691 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:21.146170 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:21.147492 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:21.148893 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:21.149570 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:21.149697 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:21.156345 (+ 6648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:21.156365 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:21.156385 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:21.156448 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:21.160689 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:21.161595 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:21.161606 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:21.161610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:21.161823 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:21.162288 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:21.162294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:21.162297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:21.162774 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:21.163141 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:21.163364 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:54:21.163598 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:54:21.164753 29431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:21.311476 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 29300
I20260620 14:54:21.331446 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 29449
I20260620 14:54:21.348416 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:21.332310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.21.159.195:43335)
0620 14:54:21.332493 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:21.332497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:21.332520 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:54:21.332729 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:21.332733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:21.332795 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:21.332878 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:21.334740 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:21.335644 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:21.337639 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:21.338232 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:21.338337 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:21.341892 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:21.341917 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:21.341934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:21.341994 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:21.345091 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:21.345963 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:21.345971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:21.345974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:21.346106 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:21.346532 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:21.346537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:21.346539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:21.347000 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:21.347508 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:21.347811 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:54:21.348046 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:54:21.349072 29572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:21.396488 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:21.395699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38166)
0620 14:54:21.396228 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:21.396324 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":312,"threads_started":1}
I20260620 14:54:21.397897 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:21.397512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32891 (local address 127.0.0.1:38176)
0620 14:54:21.397676 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:21.397758 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:54:21.470963 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 29449
I20260620 14:54:21.493439 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 28783
I20260620 14:54:21.494546 28952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:21.616832 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 28783
W20260620 14:54:21.637838 29097 connection.cc:535] server connection from 127.0.0.1:58206 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:54:21.638281 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 28886
I20260620 14:54:21.639367 29140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:21.754809 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 28886
2026-06-20T14:54:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9928 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:21 dist-test-slave-3mqx krb5kdc[29711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:21 dist-test-slave-3mqx krb5kdc[29711](info): set up 2 sockets
Jun 20 14:54:21 dist-test-slave-3mqx krb5kdc[29711](info): commencing operation
krb5kdc: starting...
W20260620 14:54:22.908389 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:54:23 dist-test-slave-3mqx krb5kdc[29711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab.
Entry for principal oryx/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab.
I20260620 14:54:23.139300 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:41935
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab
--principal=oryx/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:41935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:23.483513 29724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:23.483902 29724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:23.484076 29724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:23.491406 29724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:23.491580 29724 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:54:23.491659 29724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:23.491758 29724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:23.491866 29724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:23.491956 29724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:23.502077 29724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:41935
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.254.keytab
--principal=oryx/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:23.503515 29724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:23.504863 29724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:23.517159 29729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:23.517081 29732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:23.517118 29730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:23.518049 29724 server_base.cc:1034] running on GCE node
I20260620 14:54:23.518924 29724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:23.520516 29724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:23.521786 29724 hybrid_clock.cc:648] HybridClock initialized: now 1781967263521716 us; error 47 us; skew 500 ppm
I20260620 14:54:23.522163 29724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:23 dist-test-slave-3mqx krb5kdc[29711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967263, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:23.525522 29724 init.cc:377] Logged in from keytab as oryx/127.21.159.254@KRBTEST.COM (short username oryx)
I20260620 14:54:23.527343 29724 webserver.cc:460] Webserver started at http://127.21.159.254:45323/ using document root <none> and password file <none>
I20260620 14:54:23.528143 29724 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:23.528287 29724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:23.528633 29724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:23.531910 29724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "a321268c94b34e66a976f1cc6c63ad07"
format_stamp: "Formatted at 2026-06-20 14:54:23 on dist-test-slave-3mqx"
server_key: "13b4eca151f70ec48248c85de8dee57c"
server_key_iv: "80f5a5aaef4a96a358c376be6aeb48f1"
server_key_version: "encryptionkey@0"
I20260620 14:54:23.532752 29724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "a321268c94b34e66a976f1cc6c63ad07"
format_stamp: "Formatted at 2026-06-20 14:54:23 on dist-test-slave-3mqx"
server_key: "13b4eca151f70ec48248c85de8dee57c"
server_key_iv: "80f5a5aaef4a96a358c376be6aeb48f1"
server_key_version: "encryptionkey@0"
I20260620 14:54:23.538859 29724 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:54:23.542788 29739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:23.544370 29724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:23.544615 29724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "a321268c94b34e66a976f1cc6c63ad07"
format_stamp: "Formatted at 2026-06-20 14:54:23 on dist-test-slave-3mqx"
server_key: "13b4eca151f70ec48248c85de8dee57c"
server_key_iv: "80f5a5aaef4a96a358c376be6aeb48f1"
server_key_version: "encryptionkey@0"
I20260620 14:54:23.544909 29724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:23.560842 29724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:23.565337 29724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:23.565651 29724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:23.588325 29724 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:41935
I20260620 14:54:23.588346 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:41935 every 8 connection(s)
I20260620 14:54:23.589851 29724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:23.592028 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 29724
I20260620 14:54:23.592252 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:23.593098 22143 external_mini_cluster.cc:1428] Setting key 399ec68b7bdd24eea862e277c2f4cf56
I20260620 14:54:23.594981 29801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:54:23 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:23.609769 29801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Bootstrap starting.
I20260620 14:54:23.613265 29801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:23.614699 29801 log.cc:824] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:23.618312 29801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: No bootstrap required, opened a new log
I20260620 14:54:23.618562 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:23.595382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.21.159.254:41935)
0620 14:54:23.596248 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:23.596279 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:23.596322 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:54:23.597885 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:23.597914 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:23.598350 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:23.598709 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:23.600791 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:23.602317 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:23.604118 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:23.604827 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:23.605015 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:23.609949 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:23.609962 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:23.609985 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:23.610042 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:23.613344 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:23.614421 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:23.614428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:23.614436 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:23.614635 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:23.615168 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:23.615172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:23.615175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:23.616140 (+ 965us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:23.616427 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:23.616999 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 14:54:23.617480 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":439,"threads_started":1}
I20260620 14:54:23.626986 29801 raft_consensus.cc:348] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a321268c94b34e66a976f1cc6c63ad07" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41935 } }
I20260620 14:54:23.627528 29801 raft_consensus.cc:374] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:23.627650 29801 raft_consensus.cc:729] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a321268c94b34e66a976f1cc6c63ad07, State: Initialized, Role: FOLLOWER
I20260620 14:54:23.628352 29801 consensus_queue.cc:260] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a321268c94b34e66a976f1cc6c63ad07" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41935 } }
I20260620 14:54:23.628640 29801 raft_consensus.cc:388] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:23.628810 29801 raft_consensus.cc:482] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:23.629014 29801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:23.631289 29801 raft_consensus.cc:504] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a321268c94b34e66a976f1cc6c63ad07" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41935 } }
I20260620 14:54:23.631852 29801 leader_election.cc:302] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a321268c94b34e66a976f1cc6c63ad07; no voters:
I20260620 14:54:23.632496 29801 leader_election.cc:288] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:23.632665 29806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:23.633908 29806 raft_consensus.cc:686] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [term 1 LEADER]: Becoming Leader. State: Replica: a321268c94b34e66a976f1cc6c63ad07, State: Running, Role: LEADER
I20260620 14:54:23.634510 29806 consensus_queue.cc:237] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a321268c94b34e66a976f1cc6c63ad07" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41935 } }
I20260620 14:54:23.635255 29801 sys_catalog.cc:564] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:23.637656 29807 sys_catalog.cc:455] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a321268c94b34e66a976f1cc6c63ad07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a321268c94b34e66a976f1cc6c63ad07" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41935 } } }
I20260620 14:54:23.637845 29808 sys_catalog.cc:455] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a321268c94b34e66a976f1cc6c63ad07. Latest consensus state: current_term: 1 leader_uuid: "a321268c94b34e66a976f1cc6c63ad07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a321268c94b34e66a976f1cc6c63ad07" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41935 } } }
I20260620 14:54:23.638090 29808 sys_catalog.cc:458] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:23.638088 29807 sys_catalog.cc:458] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:23.638983 29813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:23.645030 29813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:23.654208 29813 catalog_manager.cc:1347] Generated new cluster ID: 79c5be51fe93465dba60a0e7b9aa9dbb
I20260620 14:54:23.654381 29813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:23.677533 29813 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:23.679581 29813 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:23.694612 29813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Generated new TSK 0
I20260620 14:54:23.695888 29813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab.
Entry for principal oryx/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab.
I20260620 14:54:23.751289 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41935
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab
--principal=oryx/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:24.099457 29827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:24.099771 29827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:24.099888 29827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:24.107465 29827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:24.107595 29827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:24.107681 29827 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:54:24.107760 29827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:24.118157 29827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab
--principal=oryx/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:41935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:24.119705 29827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:24.121114 29827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:24.135555 29833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:24.135600 29832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:24.135653 29835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:24.136396 29827 server_base.cc:1034] running on GCE node
I20260620 14:54:24.137105 29827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:24.138135 29827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:24.139457 29827 hybrid_clock.cc:648] HybridClock initialized: now 1781967264139381 us; error 39 us; skew 500 ppm
I20260620 14:54:24.139806 29827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:24 dist-test-slave-3mqx krb5kdc[29711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967264, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:24.143307 29827 init.cc:377] Logged in from keytab as oryx/127.21.159.193@KRBTEST.COM (short username oryx)
I20260620 14:54:24.144836 29827 webserver.cc:460] Webserver started at http://127.21.159.193:38605/ using document root <none> and password file <none>
I20260620 14:54:24.145526 29827 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:24.145612 29827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:24.145892 29827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:24.149019 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "e2a18d830bff44ba84f4e74ba33c7adf"
format_stamp: "Formatted at 2026-06-20 14:54:24 on dist-test-slave-3mqx"
server_key: "088681449884e4138ef8d2a4b5dae195"
server_key_iv: "c6e19cbbbd64654dcda1d044261256f8"
server_key_version: "encryptionkey@0"
I20260620 14:54:24.149839 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "e2a18d830bff44ba84f4e74ba33c7adf"
format_stamp: "Formatted at 2026-06-20 14:54:24 on dist-test-slave-3mqx"
server_key: "088681449884e4138ef8d2a4b5dae195"
server_key_iv: "c6e19cbbbd64654dcda1d044261256f8"
server_key_version: "encryptionkey@0"
I20260620 14:54:24.155658 29827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 14:54:24.159536 29842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:24.161134 29827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:54:24.161365 29827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "e2a18d830bff44ba84f4e74ba33c7adf"
format_stamp: "Formatted at 2026-06-20 14:54:24 on dist-test-slave-3mqx"
server_key: "088681449884e4138ef8d2a4b5dae195"
server_key_iv: "c6e19cbbbd64654dcda1d044261256f8"
server_key_version: "encryptionkey@0"
I20260620 14:54:24.161617 29827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:24.179162 29827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:24.184005 29827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:24.184288 29827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:24.187223 29827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:24.187342 29827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:24.187513 29827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:24.187623 29827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:54:24 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967264, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:24.215603 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.187827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:60047 (local address 127.21.159.254:41935)
0620 14:54:24.188551 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:24.188559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:24.189803 (+ 1244us) server_negotiation.cc:407] Connection header received
0620 14:54:24.193091 (+ 3288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:24.193099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:24.193217 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:24.193354 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:24.196348 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.197697 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.201307 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.201872 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.202064 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.207722 (+ 5658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:24.207732 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:24.207739 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:24.207783 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:24.211526 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.212999 (+ 1473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.213007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.213012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.213171 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.213692 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.213697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.213700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.213978 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:24.214139 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:24.214959 (+ 820us) server_negotiation.cc:299] Negotiation successful
0620 14:54:24.215216 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":464,"threads_started":1}
I20260620 14:54:24.217329 29852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.188599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41935 (local address 127.21.159.193:60047)
0620 14:54:24.189581 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:24.189640 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:24.192645 (+ 3005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:24.193665 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:24.193685 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:24.194822 (+ 1137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:24.196012 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.196045 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.197969 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.197978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.201057 (+ 3079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.201074 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.202293 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.202301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.202548 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.203464 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:24.203510 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:24.207332 (+ 3822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:24.211805 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.211820 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.211847 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.212809 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.213312 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.213318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.213322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.213560 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.214134 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:24.214143 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:24.214546 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 14:54:24.214849 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 14:54:24.215498 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":383,"threads_started":1}
I20260620 14:54:24.240504 29851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.224103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:33059 (local address 127.21.159.254:41935)
0620 14:54:24.224368 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:24.224376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:24.224852 (+ 476us) server_negotiation.cc:407] Connection header received
0620 14:54:24.225160 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:24.225164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:24.225229 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:24.225347 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:24.226305 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.227022 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.229256 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.229886 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.230056 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.232962 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:24.232967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:24.232970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:24.233002 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:24.236658 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.237898 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.237905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.237912 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.238098 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.238530 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.238538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.238543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.239028 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:24.239238 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:24.239976 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 14:54:24.240171 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:54:24.241093 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.224071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41935 (local address 127.21.159.193:33059)
0620 14:54:24.224716 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:24.224741 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:24.224930 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:24.225369 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:24.225374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:24.225845 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:24.226150 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.226159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.227228 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.227234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.229077 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.229101 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.230144 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.230154 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.230439 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.231298 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:24.231341 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:24.232734 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:24.236861 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.236870 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.236877 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.237702 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.238240 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.238248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.238256 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.238395 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.239332 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:24.239341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:24.239657 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 14:54:24.239879 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 14:54:24.240384 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":334,"threads_started":1}
I20260620 14:54:24.248647 29827 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:46437
I20260620 14:54:24.248661 29962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:46437 every 8 connection(s)
I20260620 14:54:24.250061 29827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:24.252355 29963 heartbeater.cc:347] Connected to a master server at 127.21.159.254:41935
I20260620 14:54:24.252605 29963 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:24.253298 29963 heartbeater.cc:511] Master 127.21.159.254:41935 requested a full tablet report, sending...
I20260620 14:54:24.255448 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 29827
I20260620 14:54:24.255666 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:24.255651 29756 ts_manager.cc:194] Registered new tserver with Master: e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437)
I20260620 14:54:24.256358 22143 external_mini_cluster.cc:1428] Setting key 22acab6eb2aece39a4d2f88e9ff0cbbf
I20260620 14:54:24.257453 29756 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:60047
WARNING: no policy specified for oryx/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab.
Entry for principal oryx/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab.
I20260620 14:54:24.307382 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41935
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab
--principal=oryx/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:24.660269 29967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:24.660576 29967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:24.660739 29967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:24.668396 29967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:24.668545 29967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:24.668699 29967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:54:24.668792 29967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:24.680109 29967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab
--principal=oryx/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:41935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:24.681545 29967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:24.682971 29967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:24.696684 29972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:24.696740 29973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:24.696713 29975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:24.697635 29967 server_base.cc:1034] running on GCE node
I20260620 14:54:24.698284 29967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:24.699371 29967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:24.700621 29967 hybrid_clock.cc:648] HybridClock initialized: now 1781967264700558 us; error 42 us; skew 500 ppm
I20260620 14:54:24.700944 29967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:24 dist-test-slave-3mqx krb5kdc[29711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967264, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:24.704530 29967 init.cc:377] Logged in from keytab as oryx/127.21.159.194@KRBTEST.COM (short username oryx)
I20260620 14:54:24.706173 29967 webserver.cc:460] Webserver started at http://127.21.159.194:33019/ using document root <none> and password file <none>
I20260620 14:54:24.706931 29967 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:24.707036 29967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:24.707449 29967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:24.710791 29967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "ba8ac39060ba40f5be2b716c67639058"
format_stamp: "Formatted at 2026-06-20 14:54:24 on dist-test-slave-3mqx"
server_key: "fec83ee985dc4968431d6891092703c6"
server_key_iv: "243f12550c3187037aaab93784f3afc1"
server_key_version: "encryptionkey@0"
I20260620 14:54:24.711825 29967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "ba8ac39060ba40f5be2b716c67639058"
format_stamp: "Formatted at 2026-06-20 14:54:24 on dist-test-slave-3mqx"
server_key: "fec83ee985dc4968431d6891092703c6"
server_key_iv: "243f12550c3187037aaab93784f3afc1"
server_key_version: "encryptionkey@0"
I20260620 14:54:24.717653 29967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:54:24.721553 29982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:24.723249 29967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:24.723451 29967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "ba8ac39060ba40f5be2b716c67639058"
format_stamp: "Formatted at 2026-06-20 14:54:24 on dist-test-slave-3mqx"
server_key: "fec83ee985dc4968431d6891092703c6"
server_key_iv: "243f12550c3187037aaab93784f3afc1"
server_key_version: "encryptionkey@0"
I20260620 14:54:24.723690 29967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:24.743431 29967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:24.748397 29967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:24.748760 29967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:24.751730 29967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:24.751819 29967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:24.751991 29967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:24.752063 29967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:54:24 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967264, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:24.782334 29992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.753465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41935 (local address 127.21.159.194:44321)
0620 14:54:24.754322 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:24.754363 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:24.756450 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:24.757724 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:24.757742 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:24.758986 (+ 1244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:24.760289 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.760333 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.762063 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.762067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.764210 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.764255 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.765757 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.765763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.766107 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.767006 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:24.767032 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:24.771095 (+ 4063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:24.775569 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.775579 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.775599 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.776377 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.777046 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.777052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.777056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.777310 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.778264 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:24.778275 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:24.778980 (+ 705us) client_negotiation.cc:769] Sending connection context
0620 14:54:24.779439 (+ 459us) client_negotiation.cc:240] Negotiation successful
0620 14:54:24.780240 (+ 801us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":361,"threads_started":1}
I20260620 14:54:24.783846 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.752676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:44321 (local address 127.21.159.254:41935)
0620 14:54:24.753160 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:24.753164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:24.754504 (+ 1340us) server_negotiation.cc:407] Connection header received
0620 14:54:24.756864 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:24.756871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:24.756982 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:24.757141 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:24.760643 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.761771 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.764473 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.765411 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.765628 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.771486 (+ 5858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:24.771494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:24.771499 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:24.771546 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:24.775325 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.776695 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.776704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.776709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.776903 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.777523 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.777530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.777536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.777990 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:24.778176 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:24.783002 (+ 4826us) server_negotiation.cc:299] Negotiation successful
0620 14:54:24.783342 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
I20260620 14:54:24.796506 29967 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:43205
I20260620 14:54:24.796597 30102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:43205 every 8 connection(s)
I20260620 14:54:24.797861 29967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:24.800158 30103 heartbeater.cc:347] Connected to a master server at 127.21.159.254:41935
I20260620 14:54:24.800441 30103 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:24.801355 30103 heartbeater.cc:511] Master 127.21.159.254:41935 requested a full tablet report, sending...
I20260620 14:54:24.803185 29755 ts_manager.cc:194] Registered new tserver with Master: ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205)
I20260620 14:54:24.804168 29755 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:44321
I20260620 14:54:24.805954 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 29967
I20260620 14:54:24.806141 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:24.806703 22143 external_mini_cluster.cc:1428] Setting key d4e214c3aff66342693742bb230d29ec
I20260620 14:54:24.808053 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.791775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:52195 (local address 127.21.159.254:41935)
0620 14:54:24.791982 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:24.791990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:24.792831 (+ 841us) server_negotiation.cc:407] Connection header received
0620 14:54:24.793164 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:24.793172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:24.793287 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:24.793411 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:24.794866 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.796200 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.797779 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.798462 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.798609 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.801096 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:24.801103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:24.801108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:24.801151 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:24.804606 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.805408 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.805431 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.805438 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.805612 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:24.806203 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:24.806208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:24.806213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:24.806630 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:24.806790 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:24.807616 (+ 826us) server_negotiation.cc:299] Negotiation successful
0620 14:54:24.807830 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:54:24.808125 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:24.791831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41935 (local address 127.21.159.194:52195)
0620 14:54:24.792683 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:24.792710 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:24.792974 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:24.793469 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:24.793474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:24.794193 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:24.794684 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.794695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.796390 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.796398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.797606 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:24.797619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:24.798707 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:24.798714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:24.798906 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:24.799862 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:24.799889 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:24.800882 (+ 993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:24.804744 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.804750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.804753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.805275 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.805849 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:24.805855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:24.805858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:24.806033 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:24.806841 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:24.806847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:24.807097 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 14:54:24.807296 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 14:54:24.807636 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":470,"threads_started":1}
WARNING: no policy specified for oryx/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab.
Entry for principal oryx/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab.
I20260620 14:54:24.856456 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:41935
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab
--principal=oryx/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:39707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:25.196266 30107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:25.196586 30107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:25.196763 30107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:25.205832 30107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:25.205964 30107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:25.206120 30107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:54:25.206236 30107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:25.216739 30107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:39707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab
--principal=oryx/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.159.254:41935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:25.218194 30107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:25.219642 30107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:25.233994 30112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:25.233911 30115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:25.233984 30113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:25.236793 30107 server_base.cc:1034] running on GCE node
I20260620 14:54:25.237534 30107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:25.238590 30107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:25.239848 30107 hybrid_clock.cc:648] HybridClock initialized: now 1781967265239759 us; error 55 us; skew 500 ppm
I20260620 14:54:25.240288 30107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:25 dist-test-slave-3mqx krb5kdc[29711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967265, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:25.243947 30107 init.cc:377] Logged in from keytab as oryx/127.21.159.195@KRBTEST.COM (short username oryx)
I20260620 14:54:25.245751 30107 webserver.cc:460] Webserver started at http://127.21.159.195:46429/ using document root <none> and password file <none>
I20260620 14:54:25.246547 30107 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:25.246649 30107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:25.246949 30107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:25.250496 30107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9"
format_stamp: "Formatted at 2026-06-20 14:54:25 on dist-test-slave-3mqx"
server_key: "4258ba792913ef653c3e16bfb342c757"
server_key_iv: "685a29a61fb0a3d368e48f556cd8144f"
server_key_version: "encryptionkey@0"
I20260620 14:54:25.251478 30107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9"
format_stamp: "Formatted at 2026-06-20 14:54:25 on dist-test-slave-3mqx"
server_key: "4258ba792913ef653c3e16bfb342c757"
server_key_iv: "685a29a61fb0a3d368e48f556cd8144f"
server_key_version: "encryptionkey@0"
I20260620 14:54:25.257464 30107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 14:54:25.260802 29963 heartbeater.cc:503] Master 127.21.159.254:41935 was elected leader, sending a full tablet report...
I20260620 14:54:25.261889 30122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:25.263863 30107 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:54:25.264070 30107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9"
format_stamp: "Formatted at 2026-06-20 14:54:25 on dist-test-slave-3mqx"
server_key: "4258ba792913ef653c3e16bfb342c757"
server_key_iv: "685a29a61fb0a3d368e48f556cd8144f"
server_key_version: "encryptionkey@0"
I20260620 14:54:25.264330 30107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:25.305653 30107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:25.311024 30107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:25.311424 30107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:25.314393 30107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:25.314483 30107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:25.314643 30107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:25.314734 30107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:54:25 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967265, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:25.346732 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.315618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:37309 (local address 127.21.159.254:41935)
0620 14:54:25.316109 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.316113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.317539 (+ 1426us) server_negotiation.cc:407] Connection header received
0620 14:54:25.320851 (+ 3312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.320860 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.320967 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.321107 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:25.324100 (+ 2993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.325390 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.329009 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.329805 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.329963 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.336412 (+ 6449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:25.336424 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:25.336432 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:25.336496 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:25.341189 (+ 4693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:25.342723 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:25.342734 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:25.342740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:25.342934 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:25.343704 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:25.343713 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:25.343719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:25.344165 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:25.344342 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.345793 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.346120 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":277,"threads_started":1}
I20260620 14:54:25.348244 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.316408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41935 (local address 127.21.159.195:37309)
0620 14:54:25.317319 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:25.317363 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:25.320406 (+ 3043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:25.321433 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:25.321451 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:25.322550 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:25.323831 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.323866 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.325613 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.325618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.328799 (+ 3181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.328815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.330090 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.330099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.330537 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.331489 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:25.331526 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:25.335978 (+ 4452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:25.341437 (+ 5459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:25.341452 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:25.341482 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:25.342497 (+ 1015us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:25.343102 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:25.343113 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:25.343117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:25.343512 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:25.344360 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:25.344378 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:25.345155 (+ 777us) client_negotiation.cc:769] Sending connection context
0620 14:54:25.345613 (+ 458us) client_negotiation.cc:240] Negotiation successful
0620 14:54:25.346418 (+ 805us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":385,"threads_started":1}
I20260620 14:54:25.362145 30107 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:34707
I20260620 14:54:25.362195 30242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:34707 every 8 connection(s)
I20260620 14:54:25.363596 30107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:25.366144 30243 heartbeater.cc:347] Connected to a master server at 127.21.159.254:41935
I20260620 14:54:25.366425 30243 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:25.367264 30243 heartbeater.cc:511] Master 127.21.159.254:41935 requested a full tablet report, sending...
I20260620 14:54:25.368878 29755 ts_manager.cc:194] Registered new tserver with Master: 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707)
I20260620 14:54:25.370090 29755 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:37309
I20260620 14:54:25.372464 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 30107
I20260620 14:54:25.372637 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:25.372838 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.356844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:46235 (local address 127.21.159.254:41935)
0620 14:54:25.357935 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.357944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.357978 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:54:25.358274 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.358282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.358373 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.358460 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:25.359970 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.360765 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.362748 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.363312 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.363473 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.366190 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:25.366198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:25.366203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:25.366248 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:25.369707 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:25.370586 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:25.370595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:25.370600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:25.370751 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:25.371213 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:25.371218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:25.371221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:25.371503 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:25.371649 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.372141 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.372389 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910}
I20260620 14:54:25.372843 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.356886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41935 (local address 127.21.159.195:46235)
0620 14:54:25.357790 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:25.357814 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:25.358010 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:25.358747 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:25.358753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:25.359386 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:25.359781 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.359795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.360931 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.360939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.362564 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.362589 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.363532 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.363539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.363786 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.364571 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:25.364595 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:25.365883 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:25.369848 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:25.369853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:25.369856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:25.370466 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:25.370869 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:25.370873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:25.370875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:25.371054 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:25.371641 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:25.371646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:25.371833 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 14:54:25.372011 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:54:25.372352 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":481,"threads_started":1}
I20260620 14:54:25.373240 22143 external_mini_cluster.cc:1428] Setting key 687290530339c54f16143c959968ed7d
I20260620 14:54:25.376518 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:54:25 dist-test-slave-3mqx krb5kdc[29711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967265, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:54:25 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:25 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:54:25.429545 30131 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:25.418295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.21.159.254:41935)
0620 14:54:25.418539 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.418547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.418803 (+ 256us) server_negotiation.cc:407] Connection header received
0620 14:54:25.419045 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.419101 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.419178 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.419289 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:25.420699 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.421434 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.422729 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.423255 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.423396 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.429216 (+ 5820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48918: BlockingRecv error: recv got EOF from 127.0.0.1:48918 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 14:54:25 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967265, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:25.448071 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.433051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.21.159.254:41935)
0620 14:54:25.433308 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.433314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.433560 (+ 246us) server_negotiation.cc:407] Connection header received
0620 14:54:25.433790 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.433796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.433877 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.433984 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:25.435219 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.436017 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.437370 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.437831 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.437968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.442316 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:25.442322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:25.442325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:25.442376 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:25.445373 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:25.446253 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:25.446258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:25.446260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:25.446411 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:25.446772 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:25.446776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:25.446778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:25.447138 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:25.447360 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.447657 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.447838 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 14:54:25.452914 29756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:25.460552 29756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:25.488464 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.473982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.21.159.195:34707)
0620 14:54:25.474798 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.474807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.475728 (+ 921us) server_negotiation.cc:407] Connection header received
0620 14:54:25.476236 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.476248 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.476584 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.476794 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:25.478095 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.480158 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.484028 (+ 3870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.486179 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.486392 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.486743 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.487683 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.487941 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":400,"threads_started":1}
I20260620 14:54:25.488873 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.474407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:43205 (local address 127.0.0.1:41056)
0620 14:54:25.477635 (+ 3228us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:25.477656 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:25.477777 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:25.479096 (+ 1319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:25.479104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:25.479137 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:25.479706 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.479731 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.482920 (+ 3189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.482929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.485470 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.485488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.487960 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.487966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.488179 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.488213 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:54:25.488298 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:54:25.488502 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3066,"thread_start_us":563,"threads_started":1}
I20260620 14:54:25.489307 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.473799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:34707 (local address 127.0.0.1:56476)
0620 14:54:25.475522 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:25.475580 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:25.475950 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:25.477249 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:25.477260 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:25.477324 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:25.477863 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.477876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.480393 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.480401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.483776 (+ 3375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.483796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.486971 (+ 3175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.486980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.487328 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.487405 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:54:25.488752 (+ 1347us) client_negotiation.cc:240] Negotiation successful
0620 14:54:25.488959 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1365,"thread_start_us":3511,"threads_started":1}
I20260620 14:54:25.490499 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.474910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.21.159.194:43205)
0620 14:54:25.478285 (+ 3375us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.478299 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.478336 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:54:25.478483 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.478493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.478818 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.480598 (+ 1780us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:25.480826 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.482727 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.485741 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.487737 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.489756 (+ 2019us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.489971 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.490047 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.490228 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3173,"thread_start_us":3316,"threads_started":1}
I20260620 14:54:25.494015 30033 tablet_service.cc:1467] Processing CreateTablet for tablet 45ad283d0a9141ccaea5a528223c25e5 (DEFAULT_TABLE table=test-table [id=6ad1fdf8708f49d8b97120c2adb8ac11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:25.494011 30173 tablet_service.cc:1467] Processing CreateTablet for tablet 45ad283d0a9141ccaea5a528223c25e5 (DEFAULT_TABLE table=test-table [id=6ad1fdf8708f49d8b97120c2adb8ac11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:25.495604 30033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ad283d0a9141ccaea5a528223c25e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:25.495620 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ad283d0a9141ccaea5a528223c25e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:25.496590 30261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.477914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46437 (local address 127.0.0.1:41068)
0620 14:54:25.489157 (+ 11243us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:25.489177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:25.489275 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:25.490093 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:25.490097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:25.490121 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:25.490285 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.490295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.492650 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.492656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.494162 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.494175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.495836 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.495842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.496020 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.496044 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:54:25.496162 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:54:25.496307 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11119,"thread_start_us":11911,"threads_started":1}
I20260620 14:54:25.496658 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.477952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41068 (local address 127.21.159.193:46437)
0620 14:54:25.489346 (+ 11394us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.489356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.489391 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:54:25.489495 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.489504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.489817 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.490085 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:25.490438 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.492508 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.494347 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.495546 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.495754 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.495945 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.496216 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.496377 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11211,"thread_start_us":11513,"threads_started":1}
I20260620 14:54:25.500088 29893 tablet_service.cc:1467] Processing CreateTablet for tablet 45ad283d0a9141ccaea5a528223c25e5 (DEFAULT_TABLE table=test-table [id=6ad1fdf8708f49d8b97120c2adb8ac11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:25.501422 29893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ad283d0a9141ccaea5a528223c25e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:25.511638 30267 tablet_bootstrap.cc:492] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Bootstrap starting.
I20260620 14:54:25.512598 30268 tablet_bootstrap.cc:492] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Bootstrap starting.
I20260620 14:54:25.514014 30269 tablet_bootstrap.cc:492] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Bootstrap starting.
I20260620 14:54:25.517355 30267 tablet_bootstrap.cc:654] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:25.517325 30268 tablet_bootstrap.cc:654] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:25.517729 30269 tablet_bootstrap.cc:654] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:25.518471 30267 log.cc:824] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:25.518740 30268 log.cc:824] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:25.518946 30269 log.cc:824] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:25.522127 30267 tablet_bootstrap.cc:492] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: No bootstrap required, opened a new log
I20260620 14:54:25.522382 30269 tablet_bootstrap.cc:492] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: No bootstrap required, opened a new log
I20260620 14:54:25.522382 30268 tablet_bootstrap.cc:492] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: No bootstrap required, opened a new log
I20260620 14:54:25.522423 30267 ts_tablet_manager.cc:1397] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260620 14:54:25.522665 30269 ts_tablet_manager.cc:1397] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:54:25.522678 30268 ts_tablet_manager.cc:1397] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260620 14:54:25.530509 30267 raft_consensus.cc:348] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.531000 30267 raft_consensus.cc:374] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:25.531122 30267 raft_consensus.cc:729] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8ac39060ba40f5be2b716c67639058, State: Initialized, Role: FOLLOWER
I20260620 14:54:25.531829 30267 consensus_queue.cc:260] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.533018 30103 heartbeater.cc:503] Master 127.21.159.254:41935 was elected leader, sending a full tablet report...
I20260620 14:54:25.533630 30267 ts_tablet_manager.cc:1428] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 14:54:25.534678 30269 raft_consensus.cc:348] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.535319 30269 raft_consensus.cc:374] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:25.535434 30269 raft_consensus.cc:729] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Initialized, Role: FOLLOWER
I20260620 14:54:25.536305 30269 consensus_queue.cc:260] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.536497 30268 raft_consensus.cc:348] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.537074 30268 raft_consensus.cc:374] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:25.537189 30268 raft_consensus.cc:729] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c4fe6164ef4adaa2f5ccbbc520d7c9, State: Initialized, Role: FOLLOWER
I20260620 14:54:25.537855 30268 consensus_queue.cc:260] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.538529 30269 ts_tablet_manager.cc:1428] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260620 14:54:25.540014 30243 heartbeater.cc:503] Master 127.21.159.254:41935 was elected leader, sending a full tablet report...
I20260620 14:54:25.540546 30268 ts_tablet_manager.cc:1428] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
W20260620 14:54:25.550271 30104 tablet.cc:2369] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:54:25.615687 30244 tablet.cc:2369] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:54:25.752625 29964 tablet.cc:2369] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:25.765854 30275 raft_consensus.cc:482] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:25.766160 30275 raft_consensus.cc:504] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.768002 30275 leader_election.cc:288] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437), ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205)
I20260620 14:54:25.774731 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.768660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:43205 (local address 127.21.159.195:40399)
0620 14:54:25.768815 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:25.768838 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:25.768979 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:25.769318 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:25.769323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:25.769357 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:25.769608 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.769620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.770900 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.770905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.772493 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.772512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.773961 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.773967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.774174 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.774198 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:54:25.774308 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:54:25.774453 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:54:25.774739 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.768574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:40399 (local address 127.21.159.194:43205)
0620 14:54:25.768805 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.768812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.768944 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:54:25.769096 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.769101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.769164 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.769295 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:25.769789 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.770765 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.772709 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.773800 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.773975 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.774056 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.774358 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.774483 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:54:25.776198 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.768635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:57959 (local address 127.21.159.193:46437)
0620 14:54:25.768806 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.768812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.769917 (+ 1105us) server_negotiation.cc:407] Connection header received
0620 14:54:25.770164 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.770170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.770260 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.770355 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:25.770946 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.771834 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.774123 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.775160 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.775368 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.775447 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.775692 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.775881 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:54:25.776226 30276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.768679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46437 (local address 127.21.159.195:57959)
0620 14:54:25.769805 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:25.769822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:25.769991 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:25.770387 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:25.770390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:25.770411 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:25.770747 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.770755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.771997 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.772005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.773942 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:25.773955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.775401 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.775407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:25.775537 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.775558 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:54:25.775634 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:54:25.775806 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":385,"thread_start_us":348,"threads_started":1}
I20260620 14:54:25.777225 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ad283d0a9141ccaea5a528223c25e5" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058" is_pre_election: true
I20260620 14:54:25.777223 29913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ad283d0a9141ccaea5a528223c25e5" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" is_pre_election: true
I20260620 14:54:25.777686 29913 raft_consensus.cc:2454] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 0.
I20260620 14:54:25.777688 30053 raft_consensus.cc:2454] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 0.
I20260620 14:54:25.778477 30125 leader_election.cc:302] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20c4fe6164ef4adaa2f5ccbbc520d7c9, ba8ac39060ba40f5be2b716c67639058, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:25.779103 30275 raft_consensus.cc:2781] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:25.779284 30275 raft_consensus.cc:482] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:25.779392 30275 raft_consensus.cc:3037] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:25.781656 30275 raft_consensus.cc:504] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.782286 30275 leader_election.cc:288] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 election: Requested vote from peers e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437), ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205)
I20260620 14:54:25.782857 29913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ad283d0a9141ccaea5a528223c25e5" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a18d830bff44ba84f4e74ba33c7adf"
I20260620 14:54:25.782866 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ad283d0a9141ccaea5a528223c25e5" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058"
I20260620 14:54:25.783099 29913 raft_consensus.cc:3037] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:25.783106 30053 raft_consensus.cc:3037] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:25.785410 30053 raft_consensus.cc:2454] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 1.
I20260620 14:54:25.785418 29913 raft_consensus.cc:2454] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 1.
I20260620 14:54:25.785913 30126 leader_election.cc:302] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c4fe6164ef4adaa2f5ccbbc520d7c9, ba8ac39060ba40f5be2b716c67639058; no voters:
I20260620 14:54:25.786269 30275 raft_consensus.cc:2781] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:25.786755 30275 raft_consensus.cc:686] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 LEADER]: Becoming Leader. State: Replica: 20c4fe6164ef4adaa2f5ccbbc520d7c9, State: Running, Role: LEADER
I20260620 14:54:25.787396 30275 consensus_queue.cc:237] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } }
I20260620 14:54:25.792169 29754 catalog_manager.cc:5495] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195). New cstate: current_term: 1 leader_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:25.857419 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.852049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.21.159.195:34707)
0620 14:54:25.852228 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:25.852232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:25.852328 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:54:25.852558 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:25.852561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:25.852639 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:25.852768 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:25.853186 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.853907 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.855356 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:25.855746 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:25.855853 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:25.856336 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:25.856881 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:25.856992 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:25.857127 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:54:25.857194 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:54:25.865612 29913 raft_consensus.cc:1261] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Refusing update from remote peer 20c4fe6164ef4adaa2f5ccbbc520d7c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:25.865648 30053 raft_consensus.cc:1261] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Refusing update from remote peer 20c4fe6164ef4adaa2f5ccbbc520d7c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:25.866559 30277 consensus_queue.cc:1035] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:25.866988 30275 consensus_queue.cc:1035] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:25.882035 30282 mvcc.cc:205] Tried to move back new op lower bound from 7298937920975855616 to 7298937920668995584. Current Snapshot: MvccSnapshot[applied={T|T < 7298937920975855616}]
I20260620 14:54:25.889851 30281 mvcc.cc:205] Tried to move back new op lower bound from 7298937920975855616 to 7298937920668995584. Current Snapshot: MvccSnapshot[applied={T|T < 7298937920975855616}]
I20260620 14:54:25.894431 29754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:25.894687 29754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:25.898300 29754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Sending DeleteTablet for 3 replicas of tablet 45ad283d0a9141ccaea5a528223c25e5
I20260620 14:54:25.899641 29893 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ad283d0a9141ccaea5a528223c25e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:25 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41068
I20260620 14:54:25.899647 30173 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ad283d0a9141ccaea5a528223c25e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:25 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56476
I20260620 14:54:25.900665 30033 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ad283d0a9141ccaea5a528223c25e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:25 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41056
I20260620 14:54:25.903407 29754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:25.904156 29754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:25.912563 30291 tablet_replica.cc:330] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: stopping tablet replica
I20260620 14:54:25.913424 30291 raft_consensus.cc:2229] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:25.914417 30291 raft_consensus.cc:2258] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:25.916013 29892 tablet_service.cc:1467] Processing CreateTablet for tablet 08b285cf535f4816a03222951b75c8f8 (DEFAULT_TABLE table=test-table [id=a0ca5d686114458c93b8ef5704e7e636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:25.916715 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b285cf535f4816a03222951b75c8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:25.919747 30032 tablet_service.cc:1467] Processing CreateTablet for tablet 08b285cf535f4816a03222951b75c8f8 (DEFAULT_TABLE table=test-table [id=a0ca5d686114458c93b8ef5704e7e636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:25.920516 30032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b285cf535f4816a03222951b75c8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:25.923851 30173 tablet_service.cc:1467] Processing CreateTablet for tablet 08b285cf535f4816a03222951b75c8f8 (DEFAULT_TABLE table=test-table [id=a0ca5d686114458c93b8ef5704e7e636]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:25.927804 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b285cf535f4816a03222951b75c8f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:25.929840 30293 tablet_replica.cc:330] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: stopping tablet replica
I20260620 14:54:25.930480 30293 raft_consensus.cc:2229] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:25.931027 30293 raft_consensus.cc:2258] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:25.936949 30269 tablet_bootstrap.cc:492] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: Bootstrap starting.
I20260620 14:54:25.938685 30269 tablet_bootstrap.cc:654] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:25.940250 30268 tablet_bootstrap.cc:492] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Bootstrap starting.
I20260620 14:54:25.940594 30293 ts_tablet_manager.cc:1905] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:25.941504 30292 tablet_replica.cc:330] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: stopping tablet replica
I20260620 14:54:25.941893 30268 tablet_bootstrap.cc:654] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:25.941998 30292 raft_consensus.cc:2229] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:25.942493 30292 raft_consensus.cc:2258] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:25.943223 30267 tablet_bootstrap.cc:492] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: Bootstrap starting.
I20260620 14:54:25.944447 30268 tablet_bootstrap.cc:492] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: No bootstrap required, opened a new log
I20260620 14:54:25.944586 30268 ts_tablet_manager.cc:1397] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 14:54:25.944886 30292 ts_tablet_manager.cc:1905] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:25.945185 30267 tablet_bootstrap.cc:654] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:25.945299 30268 raft_consensus.cc:348] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:25.945528 30268 raft_consensus.cc:374] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:25.945616 30268 raft_consensus.cc:729] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c4fe6164ef4adaa2f5ccbbc520d7c9, State: Initialized, Role: FOLLOWER
I20260620 14:54:25.946054 30268 consensus_queue.cc:260] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:25.948457 30269 tablet_bootstrap.cc:492] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: No bootstrap required, opened a new log
I20260620 14:54:25.948674 30269 ts_tablet_manager.cc:1397] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 14:54:25.949189 30268 ts_tablet_manager.cc:1428] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260620 14:54:25.949733 30269 raft_consensus.cc:348] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:25.950066 30269 raft_consensus.cc:374] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:25.950207 30269 raft_consensus.cc:729] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Initialized, Role: FOLLOWER
I20260620 14:54:25.950634 30269 consensus_queue.cc:260] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:25.951304 30269 ts_tablet_manager.cc:1428] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:54:25.952831 30292 ts_tablet_manager.cc:1918] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:25.952994 30292 log.cc:1196] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/45ad283d0a9141ccaea5a528223c25e5
I20260620 14:54:25.953219 30291 ts_tablet_manager.cc:1905] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:25.953583 30292 ts_tablet_manager.cc:1939] T 45ad283d0a9141ccaea5a528223c25e5 P e2a18d830bff44ba84f4e74ba33c7adf: Deleting consensus metadata
I20260620 14:54:25.955476 29742 catalog_manager.cc:4841] TS e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437): tablet 45ad283d0a9141ccaea5a528223c25e5 (table test-table [id=6ad1fdf8708f49d8b97120c2adb8ac11]) successfully deleted
I20260620 14:54:25.957625 30267 tablet_bootstrap.cc:492] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: No bootstrap required, opened a new log
I20260620 14:54:25.957803 30267 ts_tablet_manager.cc:1397] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20260620 14:54:25.958652 30267 raft_consensus.cc:348] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:25.958885 30267 raft_consensus.cc:374] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:25.958999 30267 raft_consensus.cc:729] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8ac39060ba40f5be2b716c67639058, State: Initialized, Role: FOLLOWER
I20260620 14:54:25.959378 30267 consensus_queue.cc:260] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:25.959514 30293 ts_tablet_manager.cc:1918] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:25.959813 30293 log.cc:1196] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/45ad283d0a9141ccaea5a528223c25e5
I20260620 14:54:25.960608 30267 ts_tablet_manager.cc:1428] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:25.961004 30293 ts_tablet_manager.cc:1939] T 45ad283d0a9141ccaea5a528223c25e5 P ba8ac39060ba40f5be2b716c67639058: Deleting consensus metadata
I20260620 14:54:25.962423 30291 ts_tablet_manager.cc:1918] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:25.962601 30291 log.cc:1196] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/45ad283d0a9141ccaea5a528223c25e5
I20260620 14:54:25.962956 29743 catalog_manager.cc:4841] TS ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205): tablet 45ad283d0a9141ccaea5a528223c25e5 (table test-table [id=6ad1fdf8708f49d8b97120c2adb8ac11]) successfully deleted
I20260620 14:54:25.963181 30291 ts_tablet_manager.cc:1939] T 45ad283d0a9141ccaea5a528223c25e5 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting consensus metadata
I20260620 14:54:25.964891 29742 catalog_manager.cc:4841] TS 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707): tablet 45ad283d0a9141ccaea5a528223c25e5 (table test-table [id=6ad1fdf8708f49d8b97120c2adb8ac11]) successfully deleted
I20260620 14:54:25.997117 30274 raft_consensus.cc:482] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:25.997380 30274 raft_consensus.cc:504] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:25.999359 30274 leader_election.cc:288] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205), 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707)
I20260620 14:54:26.006930 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.999859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:60041 (local address 127.21.159.194:43205)
0620 14:54:26.000038 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.000045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.000732 (+ 687us) server_negotiation.cc:407] Connection header received
0620 14:54:26.001031 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.001038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.001128 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.001249 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:26.001921 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.002743 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.005095 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.005858 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.006048 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.006130 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:26.006474 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:54:26.006629 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:54:26.006980 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.999789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:43205 (local address 127.21.159.193:60041)
0620 14:54:26.000599 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:26.000626 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:26.000850 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:26.001311 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:26.001318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:26.001351 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:26.001728 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.001743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.002886 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.002893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.004936 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.004952 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.006095 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.006102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.006281 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.006316 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:54:26.006423 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:54:26.006568 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":1,"thread_start_us":474,"threads_started":1}
I20260620 14:54:26.008302 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b285cf535f4816a03222951b75c8f8" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058" is_pre_election: true
I20260620 14:54:26.008559 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.000105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:45071 (local address 127.21.159.195:34707)
0620 14:54:26.000329 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.000335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.001135 (+ 800us) server_negotiation.cc:407] Connection header received
0620 14:54:26.001315 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.001320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.001412 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.001463 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:26.002202 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.003431 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.006293 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.007582 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.007773 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.007855 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:26.008129 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 14:54:26.008260 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:54:26.008562 30053 raft_consensus.cc:2454] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 0.
I20260620 14:54:26.008615 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:25.999789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:34707 (local address 127.21.159.193:45071)
0620 14:54:26.001007 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:26.001025 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:26.001147 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:26.001622 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:26.001625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:26.001644 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:26.002019 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.002032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.003611 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.003616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.006032 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.006047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.007797 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.007803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.007976 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.008000 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:54:26.008104 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:54:26.008237 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"mutex_wait_us":784,"thread_start_us":304,"threads_started":1}
I20260620 14:54:26.009244 29846 leader_election.cc:302] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8ac39060ba40f5be2b716c67639058, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:26.009681 30274 raft_consensus.cc:2781] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:26.009528 30193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b285cf535f4816a03222951b75c8f8" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" is_pre_election: true
I20260620 14:54:26.009810 30274 raft_consensus.cc:482] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:26.009894 30274 raft_consensus.cc:3037] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:26.010025 30193 raft_consensus.cc:2454] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 0.
I20260620 14:54:26.012387 30274 raft_consensus.cc:504] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.013032 30274 leader_election.cc:288] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 election: Requested vote from peers ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205), 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707)
I20260620 14:54:26.013576 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b285cf535f4816a03222951b75c8f8" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058"
I20260620 14:54:26.013600 30193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b285cf535f4816a03222951b75c8f8" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9"
I20260620 14:54:26.013796 30053 raft_consensus.cc:3037] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:26.013847 30193 raft_consensus.cc:3037] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:26.015923 30053 raft_consensus.cc:2454] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 1.
I20260620 14:54:26.016367 29846 leader_election.cc:302] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8ac39060ba40f5be2b716c67639058, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:26.016685 30274 raft_consensus.cc:2781] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:26.017230 30274 raft_consensus.cc:686] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 LEADER]: Becoming Leader. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Running, Role: LEADER
I20260620 14:54:26.017325 30193 raft_consensus.cc:2454] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 1.
I20260620 14:54:26.017642 30274 consensus_queue.cc:237] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.021405 29756 catalog_manager.cc:5495] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf reported cstate change: term changed from 0 to 1, leader changed from <none> to e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193). New cstate: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:54:26 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967263, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.254@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:26.104959 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.085495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:41935 (local address 127.0.0.1:48932)
0620 14:54:26.086703 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:26.086727 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:26.086882 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:26.087603 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:26.087607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:26.088309 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:26.088524 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.088533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.090093 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.090096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.091156 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.091168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.092133 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.092137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.092260 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.095765 (+ 3505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:26.095784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:26.098026 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:26.102200 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:26.102215 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:26.102221 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:26.102749 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:26.103180 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:26.103183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:26.103186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:26.103365 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:26.104017 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:26.104025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:26.104188 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 14:54:26.104391 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:54:26.104634 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":849,"threads_started":1}
I20260620 14:54:26.105286 30306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.085520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.21.159.254:41935)
0620 14:54:26.087240 (+ 1720us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.087246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.087268 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:54:26.087321 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.087326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.087405 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.087507 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:26.088697 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.089955 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.091286 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.091974 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.092098 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.098278 (+ 6180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:26.098286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:26.098291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:26.098330 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:26.102000 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:26.102880 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:26.102886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:26.102889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:26.103036 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:26.103479 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:26.103484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:26.103487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:26.103845 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:26.103981 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:26.104487 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 14:54:26.104822 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":1614,"thread_start_us":685,"threads_started":1}
I20260620 14:54:26.110455 29756 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:48932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:54:26.111543 29756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:26.118796 30032 tablet_service.cc:1467] Processing CreateTablet for tablet 600a6b3c70c6473db1d1a0d30d17ad44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f2029a88c1d48c0b6b007cd237dedb7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:54:26.118765 29892 tablet_service.cc:1467] Processing CreateTablet for tablet 600a6b3c70c6473db1d1a0d30d17ad44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f2029a88c1d48c0b6b007cd237dedb7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:54:26.119360 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 600a6b3c70c6473db1d1a0d30d17ad44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:26.119524 30173 tablet_service.cc:1467] Processing CreateTablet for tablet 600a6b3c70c6473db1d1a0d30d17ad44 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7f2029a88c1d48c0b6b007cd237dedb7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:54:26.120085 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 600a6b3c70c6473db1d1a0d30d17ad44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:26.125443 30268 tablet_bootstrap.cc:492] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Bootstrap starting.
I20260620 14:54:26.125698 30032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 600a6b3c70c6473db1d1a0d30d17ad44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:26.126840 30269 tablet_bootstrap.cc:492] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: Bootstrap starting.
I20260620 14:54:26.127272 30268 tablet_bootstrap.cc:654] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:26.128497 30269 tablet_bootstrap.cc:654] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:26.136689 30268 tablet_bootstrap.cc:492] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: No bootstrap required, opened a new log
I20260620 14:54:26.137039 30268 ts_tablet_manager.cc:1397] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20260620 14:54:26.138036 30268 raft_consensus.cc:348] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.138402 30268 raft_consensus.cc:374] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:26.138550 30268 raft_consensus.cc:729] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c4fe6164ef4adaa2f5ccbbc520d7c9, State: Initialized, Role: FOLLOWER
I20260620 14:54:26.138557 30267 tablet_bootstrap.cc:492] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: Bootstrap starting.
I20260620 14:54:26.138909 30268 consensus_queue.cc:260] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.139465 30287 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.139813 30287 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:54:26.140270 30267 tablet_bootstrap.cc:654] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:26.140584 30268 ts_tablet_manager.cc:1428] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:54:26.143232 30267 tablet_bootstrap.cc:492] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: No bootstrap required, opened a new log
I20260620 14:54:26.143409 30267 ts_tablet_manager.cc:1397] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 14:54:26.144243 30267 raft_consensus.cc:348] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.144493 30267 raft_consensus.cc:374] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:26.144579 30267 raft_consensus.cc:729] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8ac39060ba40f5be2b716c67639058, State: Initialized, Role: FOLLOWER
I20260620 14:54:26.144785 30269 tablet_bootstrap.cc:492] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: No bootstrap required, opened a new log
I20260620 14:54:26.144960 30269 ts_tablet_manager.cc:1397] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20260620 14:54:26.144829 30267 consensus_queue.cc:260] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.145287 30273 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.145648 30273 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:54:26.145749 30269 raft_consensus.cc:348] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.145973 30269 raft_consensus.cc:374] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:26.146055 30269 raft_consensus.cc:729] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Initialized, Role: FOLLOWER
I20260620 14:54:26.146279 30269 consensus_queue.cc:260] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.146802 30267 ts_tablet_manager.cc:1428] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:54:26.146711 30274 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.146974 30274 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:54:26.147497 30269 ts_tablet_manager.cc:1428] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:54:26.453588 30274 raft_consensus.cc:482] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:26.453853 30274 raft_consensus.cc:504] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.454542 30274 leader_election.cc:288] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205), 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707)
I20260620 14:54:26.455189 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600a6b3c70c6473db1d1a0d30d17ad44" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058" is_pre_election: true
I20260620 14:54:26.455246 30193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600a6b3c70c6473db1d1a0d30d17ad44" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" is_pre_election: true
I20260620 14:54:26.455468 30053 raft_consensus.cc:2454] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 0.
I20260620 14:54:26.455559 30193 raft_consensus.cc:2454] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 0.
I20260620 14:54:26.455976 29846 leader_election.cc:302] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20c4fe6164ef4adaa2f5ccbbc520d7c9, ba8ac39060ba40f5be2b716c67639058, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:26.456295 30274 raft_consensus.cc:2781] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:26.456401 30274 raft_consensus.cc:482] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:26.456561 30274 raft_consensus.cc:3037] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:26.458528 30274 raft_consensus.cc:504] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.459259 30274 leader_election.cc:288] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 election: Requested vote from peers ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205), 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707)
I20260620 14:54:26.459678 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600a6b3c70c6473db1d1a0d30d17ad44" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058"
I20260620 14:54:26.459946 30053 raft_consensus.cc:3037] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:26.461862 30053 raft_consensus.cc:2454] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 1.
I20260620 14:54:26.462328 29846 leader_election.cc:302] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8ac39060ba40f5be2b716c67639058, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:26.462256 30193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600a6b3c70c6473db1d1a0d30d17ad44" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9"
I20260620 14:54:26.462538 30193 raft_consensus.cc:3037] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:26.462589 30274 raft_consensus.cc:2781] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:26.462766 30274 raft_consensus.cc:686] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 LEADER]: Becoming Leader. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Running, Role: LEADER
I20260620 14:54:26.463127 30274 consensus_queue.cc:237] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.464764 30299 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: TxnStatusTablet state changed. Reason: New leader e2a18d830bff44ba84f4e74ba33c7adf. Latest consensus state: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.465071 30299 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:54:26.465279 30193 raft_consensus.cc:2454] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 1.
I20260620 14:54:26.465806 30312 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:54:26.466058 30312 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:54:26.466408 29754 catalog_manager.cc:5495] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf reported cstate change: term changed from 0 to 1, leader changed from <none> to e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193). New cstate: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:26.514794 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.508933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46437 (local address 127.0.0.1:41074)
0620 14:54:26.509254 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:26.509276 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:26.509438 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:26.509912 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:26.509917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:26.509928 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:54:26.510107 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.510116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.511171 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.511180 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.512194 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.512205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.512999 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.513005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.513190 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.513249 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:54:26.514213 (+ 964us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:54:26.514243 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:54:26.514355 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:54:26.514523 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 14:54:26.514796 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.508919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.21.159.193:46437)
0620 14:54:26.509500 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.509517 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.509546 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:54:26.509648 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.509652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.509730 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.509907 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:26.510244 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.510997 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.512349 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.512795 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.512965 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.513477 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:26.514086 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:26.514246 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:26.514376 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:54:26.514462 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":378,"threads_started":1}
I20260620 14:54:26.519330 30053 raft_consensus.cc:1261] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Refusing update from remote peer e2a18d830bff44ba84f4e74ba33c7adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:26.519524 30193 raft_consensus.cc:1261] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Refusing update from remote peer e2a18d830bff44ba84f4e74ba33c7adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:26.520212 30299 consensus_queue.cc:1035] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:26.520673 30274 consensus_queue.cc:1035] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:26.527531 30287 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: TxnStatusTablet state changed. Reason: New leader e2a18d830bff44ba84f4e74ba33c7adf. Latest consensus state: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.527825 30287 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:54:26.530421 30318 mvcc.cc:205] Tried to move back new op lower bound from 7298937923655323648 to 7298937923435126784. Current Snapshot: MvccSnapshot[applied={T|T < 7298937923655323648}]
I20260620 14:54:26.532449 30274 consensus_queue.cc:1035] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:26.535444 30287 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.535751 30287 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:54:26.542440 30273 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: TxnStatusTablet state changed. Reason: New leader e2a18d830bff44ba84f4e74ba33c7adf. Latest consensus state: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.542718 30273 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:54:26.565621 30327 consensus_queue.cc:1035] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:26.568964 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.551109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.21.159.193:46437)
0620 14:54:26.551371 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.551376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.561773 (+ 10397us) server_negotiation.cc:407] Connection header received
0620 14:54:26.562314 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.562320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.562408 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.562479 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:26.563251 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.564423 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.566701 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.567429 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.567526 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.568151 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:26.568299 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:26.568360 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:26.568585 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:54:26.568668 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:54:26.577069 30274 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.577677 30274 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:54:26.578603 30273 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.578872 30273 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P ba8ac39060ba40f5be2b716c67639058: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:54:26.580816 30299 tablet_replica.cc:439] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } } }
I20260620 14:54:26.581374 30299 tablet_replica.cc:442] T 600a6b3c70c6473db1d1a0d30d17ad44 P e2a18d830bff44ba84f4e74ba33c7adf: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:54:26.594391 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.586434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:46327 (local address 127.21.159.193:46437)
0620 14:54:26.586640 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.586644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.587409 (+ 765us) server_negotiation.cc:407] Connection header received
0620 14:54:26.587564 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.587570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.587669 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.587771 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:26.588739 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.589960 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.592046 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.592730 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.593037 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.593509 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:26.593660 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:26.593771 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:26.593922 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:54:26.594010 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:54:26.594413 30335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.586517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46437 (local address 127.21.159.193:46327)
0620 14:54:26.587298 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:26.587314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:26.587441 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:26.588145 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:26.588150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:26.588161 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:54:26.588576 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.588587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.590105 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.590111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.591897 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:26.591915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.593059 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.593066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:26.593235 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.593313 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:54:26.593776 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:54:26.593788 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:54:26.593884 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:54:26.594031 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":437,"threads_started":1}
I20260620 14:54:26.670200 29754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:26.670498 29754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:26.673679 29754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Sending DeleteTablet for 3 replicas of tablet 08b285cf535f4816a03222951b75c8f8
I20260620 14:54:26.674661 30032 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b285cf535f4816a03222951b75c8f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:26 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41056
I20260620 14:54:26.674690 30173 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b285cf535f4816a03222951b75c8f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:26 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56476
I20260620 14:54:26.674888 29892 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b285cf535f4816a03222951b75c8f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:26 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41068
I20260620 14:54:26.681897 30338 tablet_replica.cc:330] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: stopping tablet replica
I20260620 14:54:26.682071 30337 tablet_replica.cc:330] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: stopping tablet replica
I20260620 14:54:26.682276 30337 raft_consensus.cc:2229] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:26.682358 30339 tablet_replica.cc:330] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: stopping tablet replica
I20260620 14:54:26.682518 30337 raft_consensus.cc:2258] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:26.682633 30338 raft_consensus.cc:2229] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:26.682627 30339 raft_consensus.cc:2229] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:26.682905 30338 raft_consensus.cc:2258] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:26.683275 30339 raft_consensus.cc:2258] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:26.683768 30337 ts_tablet_manager.cc:1905] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:26.684092 30338 ts_tablet_manager.cc:1905] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:26.684450 30339 ts_tablet_manager.cc:1905] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:26.691354 30337 ts_tablet_manager.cc:1918] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:54:26.691354 30338 ts_tablet_manager.cc:1918] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:54:26.691504 30337 log.cc:1196] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/08b285cf535f4816a03222951b75c8f8
I20260620 14:54:26.691682 30338 log.cc:1196] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/08b285cf535f4816a03222951b75c8f8
I20260620 14:54:26.691984 30337 ts_tablet_manager.cc:1939] T 08b285cf535f4816a03222951b75c8f8 P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting consensus metadata
I20260620 14:54:26.692301 30338 ts_tablet_manager.cc:1939] T 08b285cf535f4816a03222951b75c8f8 P ba8ac39060ba40f5be2b716c67639058: Deleting consensus metadata
I20260620 14:54:26.692497 30339 ts_tablet_manager.cc:1918] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:54:26.692657 30339 log.cc:1196] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/08b285cf535f4816a03222951b75c8f8
I20260620 14:54:26.693210 30339 ts_tablet_manager.cc:1939] T 08b285cf535f4816a03222951b75c8f8 P e2a18d830bff44ba84f4e74ba33c7adf: Deleting consensus metadata
I20260620 14:54:26.693246 29742 catalog_manager.cc:4841] TS 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707): tablet 08b285cf535f4816a03222951b75c8f8 (table test-table [id=a0ca5d686114458c93b8ef5704e7e636]) successfully deleted
I20260620 14:54:26.693512 29743 catalog_manager.cc:4841] TS ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205): tablet 08b285cf535f4816a03222951b75c8f8 (table test-table [id=a0ca5d686114458c93b8ef5704e7e636]) successfully deleted
I20260620 14:54:26.694350 29742 catalog_manager.cc:4841] TS e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437): tablet 08b285cf535f4816a03222951b75c8f8 (table test-table [id=a0ca5d686114458c93b8ef5704e7e636]) successfully deleted
Jun 20 14:54:26 dist-test-slave-3mqx krb5kdc[29711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:54:26 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:26 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:54:26.743976 30347 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:26.732472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.21.159.254:41935)
0620 14:54:26.733317 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.733323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.733339 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:54:26.733404 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.733407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.733465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.733588 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:26.734920 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.735850 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.737247 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.737678 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.737831 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.743634 (+ 5803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48942: BlockingRecv error: recv got EOF from 127.0.0.1:48942 (error 108)
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":521,"threads_started":1}
Jun 20 14:54:26 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM
I20260620 14:54:26.762864 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:26.747184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.21.159.254:41935)
0620 14:54:26.747513 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:26.747522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:26.747791 (+ 269us) server_negotiation.cc:407] Connection header received
0620 14:54:26.748009 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:26.748015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:26.748090 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:26.748213 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:26.749549 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.750391 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.751840 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:26.752295 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:26.752476 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:26.757120 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:26.757126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:26.757129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:26.757170 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:26.760095 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:26.760865 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:26.760875 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:26.760878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:26.761012 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:26.761430 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:26.761440 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:26.761442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:26.761715 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:26.761991 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:26.762295 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 14:54:26.762536 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:54:26.766954 29754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:26.767781 29754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:26.775667 30032 tablet_service.cc:1467] Processing CreateTablet for tablet 95bbd582f357499a860c801e76bf208b (DEFAULT_TABLE table=test-table [id=d26bd2fa0efa47e38b1d2ebf9984a614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:26.775851 30173 tablet_service.cc:1467] Processing CreateTablet for tablet 95bbd582f357499a860c801e76bf208b (DEFAULT_TABLE table=test-table [id=d26bd2fa0efa47e38b1d2ebf9984a614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:26.776087 29892 tablet_service.cc:1467] Processing CreateTablet for tablet 95bbd582f357499a860c801e76bf208b (DEFAULT_TABLE table=test-table [id=d26bd2fa0efa47e38b1d2ebf9984a614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:26.776252 30032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bbd582f357499a860c801e76bf208b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:26.776316 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bbd582f357499a860c801e76bf208b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:26.776551 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95bbd582f357499a860c801e76bf208b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:26.788043 30354 tablet_bootstrap.cc:492] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: Bootstrap starting.
I20260620 14:54:26.789161 30355 tablet_bootstrap.cc:492] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: Bootstrap starting.
I20260620 14:54:26.789418 30353 tablet_bootstrap.cc:492] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Bootstrap starting.
I20260620 14:54:26.790150 30354 tablet_bootstrap.cc:654] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:26.791126 30353 tablet_bootstrap.cc:654] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:26.793267 30355 tablet_bootstrap.cc:654] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:26.796460 30354 tablet_bootstrap.cc:492] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: No bootstrap required, opened a new log
I20260620 14:54:26.796638 30354 ts_tablet_manager.cc:1397] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 14:54:26.796656 30353 tablet_bootstrap.cc:492] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: No bootstrap required, opened a new log
I20260620 14:54:26.796792 30353 ts_tablet_manager.cc:1397] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 14:54:26.797508 30353 raft_consensus.cc:348] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.797520 30354 raft_consensus.cc:348] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.797746 30353 raft_consensus.cc:374] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:26.797806 30354 raft_consensus.cc:374] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:26.797819 30353 raft_consensus.cc:729] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c4fe6164ef4adaa2f5ccbbc520d7c9, State: Initialized, Role: FOLLOWER
I20260620 14:54:26.797909 30354 raft_consensus.cc:729] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8ac39060ba40f5be2b716c67639058, State: Initialized, Role: FOLLOWER
I20260620 14:54:26.798121 30353 consensus_queue.cc:260] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.798164 30354 consensus_queue.cc:260] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.798754 30354 ts_tablet_manager.cc:1428] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 14:54:26.799908 30353 ts_tablet_manager.cc:1428] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:54:26.813426 30355 tablet_bootstrap.cc:492] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: No bootstrap required, opened a new log
I20260620 14:54:26.813609 30355 ts_tablet_manager.cc:1397] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20260620 14:54:26.814419 30355 raft_consensus.cc:348] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.814692 30355 raft_consensus.cc:374] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:26.814785 30355 raft_consensus.cc:729] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Initialized, Role: FOLLOWER
I20260620 14:54:26.815071 30355 consensus_queue.cc:260] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:26.815804 30355 ts_tablet_manager.cc:1428] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:54:27.024937 30287 raft_consensus.cc:482] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:27.025156 30287 raft_consensus.cc:504] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.026005 30287 leader_election.cc:288] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437), ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205)
I20260620 14:54:27.026614 29913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd582f357499a860c801e76bf208b" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" is_pre_election: true
I20260620 14:54:27.026629 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd582f357499a860c801e76bf208b" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058" is_pre_election: true
I20260620 14:54:27.026930 29913 raft_consensus.cc:2454] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 0.
I20260620 14:54:27.026966 30053 raft_consensus.cc:2454] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 0.
I20260620 14:54:27.027485 30125 leader_election.cc:302] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c4fe6164ef4adaa2f5ccbbc520d7c9, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:27.027926 30287 raft_consensus.cc:2781] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:27.028048 30287 raft_consensus.cc:482] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:27.028141 30287 raft_consensus.cc:3037] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:27.030647 30287 raft_consensus.cc:504] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.031293 30287 leader_election.cc:288] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 election: Requested vote from peers e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437), ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205)
I20260620 14:54:27.031831 29913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd582f357499a860c801e76bf208b" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a18d830bff44ba84f4e74ba33c7adf"
I20260620 14:54:27.031857 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95bbd582f357499a860c801e76bf208b" candidate_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058"
I20260620 14:54:27.032076 29913 raft_consensus.cc:3037] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:27.032102 30053 raft_consensus.cc:3037] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:27.034495 29913 raft_consensus.cc:2454] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 1.
I20260620 14:54:27.035113 30125 leader_election.cc:302] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20c4fe6164ef4adaa2f5ccbbc520d7c9, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:27.035446 30287 raft_consensus.cc:2781] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:27.035468 30053 raft_consensus.cc:2454] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20c4fe6164ef4adaa2f5ccbbc520d7c9 in term 1.
I20260620 14:54:27.039386 30287 raft_consensus.cc:686] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 LEADER]: Becoming Leader. State: Replica: 20c4fe6164ef4adaa2f5ccbbc520d7c9, State: Running, Role: LEADER
I20260620 14:54:27.039762 30287 consensus_queue.cc:237] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.043007 29755 catalog_manager.cc:5495] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195). New cstate: current_term: 1 leader_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } health_report { overall_health: HEALTHY } } }
I20260620 14:54:27.099354 30362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:27.092695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.21.159.195:34707)
0620 14:54:27.093279 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:27.093283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:27.093303 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:54:27.093361 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:27.093365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:27.093437 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:27.093524 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:27.094000 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.094815 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.097151 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.097641 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.097768 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:27.098379 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:27.098515 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:27.098598 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:27.098809 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:54:27.098900 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":335,"threads_started":1}
I20260620 14:54:27.102314 29913 raft_consensus.cc:1261] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Refusing update from remote peer 20c4fe6164ef4adaa2f5ccbbc520d7c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:27.102525 30053 raft_consensus.cc:1261] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Refusing update from remote peer 20c4fe6164ef4adaa2f5ccbbc520d7c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:27.103025 30359 consensus_queue.cc:1035] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:27.103384 30287 consensus_queue.cc:1035] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:27.116447 29756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:27.116756 29756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:27.122567 29756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Sending DeleteTablet for 3 replicas of tablet 95bbd582f357499a860c801e76bf208b
I20260620 14:54:27.123697 30173 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bbd582f357499a860c801e76bf208b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:27 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56476
I20260620 14:54:27.123912 30172 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bbd582f357499a860c801e76bf208b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:27 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56476
I20260620 14:54:27.124198 30337 tablet_replica.cc:330] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: stopping tablet replica
I20260620 14:54:27.124370 30337 raft_consensus.cc:2229] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:27.124701 30337 raft_consensus.cc:2258] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:27.124687 29756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:27.125376 29756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:27.125819 29892 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bbd582f357499a860c801e76bf208b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:27 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41068
I20260620 14:54:27.125936 30337 ts_tablet_manager.cc:1905] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:27.126799 30339 tablet_replica.cc:330] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: stopping tablet replica
I20260620 14:54:27.127003 30339 raft_consensus.cc:2229] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:27.127429 30339 raft_consensus.cc:2258] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:27.129513 30032 tablet_service.cc:1514] Processing DeleteTablet for tablet 95bbd582f357499a860c801e76bf208b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:27 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41056
I20260620 14:54:27.129824 30338 tablet_replica.cc:330] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: stopping tablet replica
I20260620 14:54:27.130030 30338 raft_consensus.cc:2229] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:27.130275 30338 raft_consensus.cc:2258] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:27.131469 30339 ts_tablet_manager.cc:1905] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:27.134049 30337 ts_tablet_manager.cc:1918] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:27.134215 30337 log.cc:1196] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/95bbd582f357499a860c801e76bf208b
I20260620 14:54:27.134922 30337 ts_tablet_manager.cc:1939] T 95bbd582f357499a860c801e76bf208b P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting consensus metadata
I20260620 14:54:27.136734 30338 ts_tablet_manager.cc:1905] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:27.136763 29742 catalog_manager.cc:4841] TS 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707): tablet 95bbd582f357499a860c801e76bf208b (table test-table [id=d26bd2fa0efa47e38b1d2ebf9984a614]) successfully deleted
W20260620 14:54:27.137440 29742 catalog_manager.cc:4808] TS 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707): delete failed for tablet 95bbd582f357499a860c801e76bf208b because the tablet was not found. No further retry: Not found: Tablet not found: 95bbd582f357499a860c801e76bf208b
I20260620 14:54:27.142160 30032 tablet_service.cc:1467] Processing CreateTablet for tablet e97558b5655b49f88730ddd0569776be (DEFAULT_TABLE table=test-table [id=08f24fd0a04e425b8874d19278a891fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:27.142599 30032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97558b5655b49f88730ddd0569776be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:27.144276 30339 ts_tablet_manager.cc:1918] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:27.144464 30339 log.cc:1196] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/95bbd582f357499a860c801e76bf208b
I20260620 14:54:27.144932 30339 ts_tablet_manager.cc:1939] T 95bbd582f357499a860c801e76bf208b P e2a18d830bff44ba84f4e74ba33c7adf: Deleting consensus metadata
I20260620 14:54:27.146183 29742 catalog_manager.cc:4841] TS e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437): tablet 95bbd582f357499a860c801e76bf208b (table test-table [id=d26bd2fa0efa47e38b1d2ebf9984a614]) successfully deleted
I20260620 14:54:27.147446 29892 tablet_service.cc:1467] Processing CreateTablet for tablet e97558b5655b49f88730ddd0569776be (DEFAULT_TABLE table=test-table [id=08f24fd0a04e425b8874d19278a891fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:27.147544 30172 tablet_service.cc:1467] Processing CreateTablet for tablet e97558b5655b49f88730ddd0569776be (DEFAULT_TABLE table=test-table [id=08f24fd0a04e425b8874d19278a891fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:27.147889 29892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97558b5655b49f88730ddd0569776be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:27.148036 30172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e97558b5655b49f88730ddd0569776be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:27.153568 30355 tablet_bootstrap.cc:492] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: Bootstrap starting.
I20260620 14:54:27.155501 30354 tablet_bootstrap.cc:492] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: Bootstrap starting.
I20260620 14:54:27.155788 30355 tablet_bootstrap.cc:654] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:27.156941 30353 tablet_bootstrap.cc:492] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Bootstrap starting.
I20260620 14:54:27.157369 30354 tablet_bootstrap.cc:654] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:27.158396 30338 ts_tablet_manager.cc:1918] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:27.158576 30338 log.cc:1196] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/95bbd582f357499a860c801e76bf208b
I20260620 14:54:27.158679 30353 tablet_bootstrap.cc:654] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:27.159112 30338 ts_tablet_manager.cc:1939] T 95bbd582f357499a860c801e76bf208b P ba8ac39060ba40f5be2b716c67639058: Deleting consensus metadata
I20260620 14:54:27.160172 29743 catalog_manager.cc:4841] TS ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205): tablet 95bbd582f357499a860c801e76bf208b (table test-table [id=d26bd2fa0efa47e38b1d2ebf9984a614]) successfully deleted
I20260620 14:54:27.161795 30353 tablet_bootstrap.cc:492] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: No bootstrap required, opened a new log
I20260620 14:54:27.162009 30353 ts_tablet_manager.cc:1397] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:54:27.163755 30353 raft_consensus.cc:348] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.164134 30353 raft_consensus.cc:374] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:27.164225 30353 raft_consensus.cc:729] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c4fe6164ef4adaa2f5ccbbc520d7c9, State: Initialized, Role: FOLLOWER
I20260620 14:54:27.164579 30353 consensus_queue.cc:260] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.166808 30355 tablet_bootstrap.cc:492] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: No bootstrap required, opened a new log
I20260620 14:54:27.167037 30355 ts_tablet_manager.cc:1397] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260620 14:54:27.168444 30355 raft_consensus.cc:348] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.168905 30355 raft_consensus.cc:374] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:27.169015 30355 raft_consensus.cc:729] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Initialized, Role: FOLLOWER
I20260620 14:54:27.169705 30355 consensus_queue.cc:260] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.170588 30353 ts_tablet_manager.cc:1428] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260620 14:54:27.171370 30355 ts_tablet_manager.cc:1428] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260620 14:54:27.173406 30354 tablet_bootstrap.cc:492] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: No bootstrap required, opened a new log
I20260620 14:54:27.173594 30354 ts_tablet_manager.cc:1397] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20260620 14:54:27.174422 30354 raft_consensus.cc:348] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.174649 30354 raft_consensus.cc:374] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:27.174731 30354 raft_consensus.cc:729] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8ac39060ba40f5be2b716c67639058, State: Initialized, Role: FOLLOWER
I20260620 14:54:27.174975 30354 consensus_queue.cc:260] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.175668 30354 ts_tablet_manager.cc:1428] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:54:27.225723 30326 raft_consensus.cc:482] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:27.226003 30326 raft_consensus.cc:504] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.226903 30326 leader_election.cc:288] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205), 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707)
I20260620 14:54:27.227443 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97558b5655b49f88730ddd0569776be" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058" is_pre_election: true
I20260620 14:54:27.227734 30053 raft_consensus.cc:2454] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 0.
I20260620 14:54:27.227746 30193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97558b5655b49f88730ddd0569776be" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" is_pre_election: true
I20260620 14:54:27.228044 30193 raft_consensus.cc:2454] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 0.
I20260620 14:54:27.228248 29846 leader_election.cc:302] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8ac39060ba40f5be2b716c67639058, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:27.228639 30326 raft_consensus.cc:2781] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:27.228760 30326 raft_consensus.cc:482] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:27.228883 30326 raft_consensus.cc:3037] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:27.231245 30326 raft_consensus.cc:504] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.231964 30326 leader_election.cc:288] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 election: Requested vote from peers ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205), 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707)
I20260620 14:54:27.232411 30053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97558b5655b49f88730ddd0569776be" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8ac39060ba40f5be2b716c67639058"
I20260620 14:54:27.232686 30053 raft_consensus.cc:3037] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:27.232707 30193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e97558b5655b49f88730ddd0569776be" candidate_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9"
I20260620 14:54:27.232966 30193 raft_consensus.cc:3037] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:27.235167 30053 raft_consensus.cc:2454] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 1.
I20260620 14:54:27.235728 29846 leader_election.cc:302] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba8ac39060ba40f5be2b716c67639058, e2a18d830bff44ba84f4e74ba33c7adf; no voters:
I20260620 14:54:27.236060 30326 raft_consensus.cc:2781] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:27.236375 30193 raft_consensus.cc:2454] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2a18d830bff44ba84f4e74ba33c7adf in term 1.
I20260620 14:54:27.236690 30326 raft_consensus.cc:686] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 1 LEADER]: Becoming Leader. State: Replica: e2a18d830bff44ba84f4e74ba33c7adf, State: Running, Role: LEADER
I20260620 14:54:27.237255 30326 consensus_queue.cc:237] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } }
I20260620 14:54:27.240298 29756 catalog_manager.cc:5495] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf reported cstate change: term changed from 0 to 1, leader changed from <none> to e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193). New cstate: current_term: 1 leader_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2a18d830bff44ba84f4e74ba33c7adf" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:27.272105 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:27.264493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.21.159.193:46437)
0620 14:54:27.265066 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:27.265072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:27.265102 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:54:27.265164 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:27.265169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:27.265251 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:27.265343 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:27.265856 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.267181 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.269408 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.270239 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.270407 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:27.271020 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:27.271242 (+ 222us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:27.271353 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:27.271564 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:54:27.271662 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":636,"threads_started":1}
I20260620 14:54:27.284312 30192 raft_consensus.cc:1261] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Refusing update from remote peer e2a18d830bff44ba84f4e74ba33c7adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:27.285041 30326 consensus_queue.cc:1035] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c4fe6164ef4adaa2f5ccbbc520d7c9" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:27.293749 30053 raft_consensus.cc:1261] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Refusing update from remote peer e2a18d830bff44ba84f4e74ba33c7adf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:27.294515 30369 consensus_queue.cc:1035] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8ac39060ba40f5be2b716c67639058" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:27.357964 29756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:27.358250 29756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:27.361840 29756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a321268c94b34e66a976f1cc6c63ad07: Sending DeleteTablet for 3 replicas of tablet e97558b5655b49f88730ddd0569776be
I20260620 14:54:27.362896 30032 tablet_service.cc:1514] Processing DeleteTablet for tablet e97558b5655b49f88730ddd0569776be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:27 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41056
I20260620 14:54:27.363282 30338 tablet_replica.cc:330] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: stopping tablet replica
I20260620 14:54:27.363526 30338 raft_consensus.cc:2229] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:27.363788 30338 raft_consensus.cc:2258] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:27.364679 30172 tablet_service.cc:1514] Processing DeleteTablet for tablet e97558b5655b49f88730ddd0569776be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:27 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56476
I20260620 14:54:27.365063 30337 tablet_replica.cc:330] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: stopping tablet replica
I20260620 14:54:27.365270 30337 raft_consensus.cc:2229] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:27.365391 29892 tablet_service.cc:1514] Processing DeleteTablet for tablet e97558b5655b49f88730ddd0569776be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:27 UTC) from {username='oryx', principal='oryx/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:41068
I20260620 14:54:27.365497 30337 raft_consensus.cc:2258] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:27.365808 30339 tablet_replica.cc:330] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: stopping tablet replica
I20260620 14:54:27.366034 30339 raft_consensus.cc:2229] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:27.366422 30339 raft_consensus.cc:2258] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:27.366534 30338 ts_tablet_manager.cc:1905] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:27.367733 30339 ts_tablet_manager.cc:1905] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:27.371892 30337 ts_tablet_manager.cc:1905] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:27.376762 30339 ts_tablet_manager.cc:1918] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:54:27.377112 30339 log.cc:1196] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/e97558b5655b49f88730ddd0569776be
I20260620 14:54:27.377785 30339 ts_tablet_manager.cc:1939] T e97558b5655b49f88730ddd0569776be P e2a18d830bff44ba84f4e74ba33c7adf: Deleting consensus metadata
I20260620 14:54:27.377861 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 29827
I20260620 14:54:27.379478 29742 catalog_manager.cc:4841] TS e2a18d830bff44ba84f4e74ba33c7adf (127.21.159.193:46437): tablet e97558b5655b49f88730ddd0569776be (table test-table [id=08f24fd0a04e425b8874d19278a891fd]) successfully deleted
I20260620 14:54:27.384461 30337 ts_tablet_manager.cc:1918] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:54:27.384822 30337 log.cc:1196] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/e97558b5655b49f88730ddd0569776be
I20260620 14:54:27.385596 30337 ts_tablet_manager.cc:1939] T e97558b5655b49f88730ddd0569776be P 20c4fe6164ef4adaa2f5ccbbc520d7c9: Deleting consensus metadata
I20260620 14:54:27.387528 29742 catalog_manager.cc:4841] TS 20c4fe6164ef4adaa2f5ccbbc520d7c9 (127.21.159.195:34707): tablet e97558b5655b49f88730ddd0569776be (table test-table [id=08f24fd0a04e425b8874d19278a891fd]) successfully deleted
I20260620 14:54:27.388787 30338 ts_tablet_manager.cc:1918] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:54:27.389010 30338 log.cc:1196] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/e97558b5655b49f88730ddd0569776be
I20260620 14:54:27.389590 30338 ts_tablet_manager.cc:1939] T e97558b5655b49f88730ddd0569776be P ba8ac39060ba40f5be2b716c67639058: Deleting consensus metadata
I20260620 14:54:27.391038 29743 catalog_manager.cc:4841] TS ba8ac39060ba40f5be2b716c67639058 (127.21.159.194:43205): tablet e97558b5655b49f88730ddd0569776be (table test-table [id=08f24fd0a04e425b8874d19278a891fd]) successfully deleted
Jun 20 14:54:27 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.193@KRBTEST.COM
I20260620 14:54:27.407869 30371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:27.379481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.21.159.193:46437)
0620 14:54:27.383401 (+ 3920us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:27.383410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:27.387883 (+ 4473us) server_negotiation.cc:407] Connection header received
0620 14:54:27.388240 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:27.388251 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:27.388414 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:27.388526 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:27.391132 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.392369 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.394044 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.394618 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.394832 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:27.399902 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:27.399918 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:27.399935 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:27.399987 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:27.404070 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:27.405008 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:27.405018 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:27.405021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:27.405237 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:27.405726 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:27.405734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:27.405738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:27.406183 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:27.406574 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:27.406986 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 14:54:27.407314 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3690}
I20260620 14:54:27.408785 29958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:27.567981 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 29827
I20260620 14:54:27.595314 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 29967
Jun 20 14:54:27 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.194@KRBTEST.COM
I20260620 14:54:27.616178 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:27.596268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.21.159.194:43205)
0620 14:54:27.596884 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:27.596890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:27.596907 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:54:27.596970 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:27.596974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:27.597045 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:27.597144 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:27.598932 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.599900 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.601880 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.602352 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.602460 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:27.608492 (+ 6032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:27.608517 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:27.608539 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:27.608608 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:27.612792 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:27.613665 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:27.613676 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:27.613681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:27.613861 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:27.614252 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:27.614258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:27.614262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:27.614748 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:27.615115 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:27.615456 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 14:54:27.615704 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":401,"threads_started":1}
I20260620 14:54:27.616926 30098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:27.716897 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 29967
I20260620 14:54:27.736300 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 30107
Jun 20 14:54:27 dist-test-slave-3mqx krb5kdc[29711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.159.195@KRBTEST.COM
I20260620 14:54:27.753783 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:27.737208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.21.159.195:34707)
0620 14:54:27.737659 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:27.737664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:27.737681 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:54:27.737736 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:27.737740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:27.737801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:27.737902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:27.739419 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.740171 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.741518 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:27.742008 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:27.742116 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:27.747230 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:27.747248 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:27.747267 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:27.747308 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:27.750334 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:27.751359 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:27.751367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:27.751370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:27.751512 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:27.751960 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:27.751963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:27.751965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:27.752404 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:27.752658 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:27.753149 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 14:54:27.753387 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":271,"threads_started":1}
I20260620 14:54:27.754587 30238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:27.881637 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 30107
I20260620 14:54:27.903802 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 29724
I20260620 14:54:27.905170 29796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:28.021220 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 29724
2026-06-20T14:54:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6265 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:28 dist-test-slave-3mqx krb5kdc[30387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:28 dist-test-slave-3mqx krb5kdc[30387](info): set up 2 sockets
Jun 20 14:54:28 dist-test-slave-3mqx krb5kdc[30387](info): commencing operation
krb5kdc: starting...
W20260620 14:54:29.217110 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:54:29 dist-test-slave-3mqx krb5kdc[30387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:54:29.447321 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:40445
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:40445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:29.796209 30400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:29.796540 30400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:29.796708 30400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:29.804147 30400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:29.804276 30400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:29.804368 30400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:29.804452 30400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:29.804543 30400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:29.814709 30400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:40445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:29.816128 30400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:29.817600 30400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:29.830556 30406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:29.830791 30405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:29.830569 30408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:29.831502 30400 server_base.cc:1034] running on GCE node
I20260620 14:54:29.832352 30400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:29.833885 30400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:29.835227 30400 hybrid_clock.cc:648] HybridClock initialized: now 1781967269835171 us; error 65 us; skew 500 ppm
I20260620 14:54:29.835638 30400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:54:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1883 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:30 dist-test-slave-3mqx krb5kdc[30414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:30 dist-test-slave-3mqx krb5kdc[30414](info): set up 2 sockets
Jun 20 14:54:30 dist-test-slave-3mqx krb5kdc[30414](info): commencing operation
krb5kdc: starting...
W20260620 14:54:31.012280 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:54:31 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:54:31.248680 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38017
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:31.608321 30427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:31.608734 30427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:31.608923 30427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:31.616478 30427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:31.616671 30427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:31.616824 30427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:31.616928 30427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:31.617020 30427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:31.627245 30427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:31.628749 30427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:31.630170 30427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:31.642755 30435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:31.642887 30432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:31.642776 30433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:31.643703 30427 server_base.cc:1034] running on GCE node
I20260620 14:54:31.644548 30427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:31.646001 30427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:31.647245 30427 hybrid_clock.cc:648] HybridClock initialized: now 1781967271647161 us; error 57 us; skew 500 ppm
I20260620 14:54:31.647652 30427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:31 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967271, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:31.650926 30427 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:54:31.652606 30427 webserver.cc:460] Webserver started at http://127.21.159.254:33763/ using document root <none> and password file <none>
I20260620 14:54:31.653369 30427 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:31.653515 30427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:31.653837 30427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:31.657087 30427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "1e4b4bf55ed040ae8799270c5b3ed33d"
format_stamp: "Formatted at 2026-06-20 14:54:31 on dist-test-slave-3mqx"
server_key: "42b3ce1ba630c673e17f7bce28894e72"
server_key_iv: "0058d5f4c4ec2c03098eb42b4a94bae0"
server_key_version: "encryptionkey@0"
I20260620 14:54:31.657905 30427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "1e4b4bf55ed040ae8799270c5b3ed33d"
format_stamp: "Formatted at 2026-06-20 14:54:31 on dist-test-slave-3mqx"
server_key: "42b3ce1ba630c673e17f7bce28894e72"
server_key_iv: "0058d5f4c4ec2c03098eb42b4a94bae0"
server_key_version: "encryptionkey@0"
I20260620 14:54:31.663578 30427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 14:54:31.667431 30442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:31.668942 30427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:54:31.669168 30427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "1e4b4bf55ed040ae8799270c5b3ed33d"
format_stamp: "Formatted at 2026-06-20 14:54:31 on dist-test-slave-3mqx"
server_key: "42b3ce1ba630c673e17f7bce28894e72"
server_key_iv: "0058d5f4c4ec2c03098eb42b4a94bae0"
server_key_version: "encryptionkey@0"
I20260620 14:54:31.669394 30427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:31.690295 30427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:31.695176 30427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:31.695467 30427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:31.715430 30427 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:38017
I20260620 14:54:31.715461 30493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:38017 every 8 connection(s)
I20260620 14:54:31.716892 30427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:31.720832 30494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:31.724464 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 30427
I20260620 14:54:31.724702 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:31.725499 22143 external_mini_cluster.cc:1428] Setting key 6899e4318c1aec59cb5551e402a36458
I20260620 14:54:31.734259 30494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Bootstrap starting.
I20260620 14:54:31.738114 30494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:31.739356 30494 log.cc:824] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:54:31 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:31.743161 30494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: No bootstrap required, opened a new log
I20260620 14:54:31.751479 30497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:31.727851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47028 (local address 127.21.159.254:38017)
0620 14:54:31.728617 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:31.728630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:31.728674 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:54:31.730369 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:31.730394 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:31.730844 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:31.731320 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:31.732859 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:31.734217 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:31.735627 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:31.736359 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:31.736568 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:31.741299 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:31.741311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:31.741334 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:31.741384 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:31.745658 (+ 4274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:31.746525 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:31.746540 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:31.746553 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:31.746794 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:31.747414 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:31.747420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:31.747424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:31.748380 (+ 956us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:31.748727 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:31.749250 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 14:54:31.749789 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":325,"threads_started":1}
I20260620 14:54:31.754285 30494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:31.754777 30494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:31.754944 30494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4b4bf55ed040ae8799270c5b3ed33d, State: Initialized, Role: FOLLOWER
I20260620 14:54:31.755611 30494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:31.755901 30494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:31.756069 30494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:31.756278 30494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:31.758714 30494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:31.759455 30494 leader_election.cc:302] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e4b4bf55ed040ae8799270c5b3ed33d; no voters:
I20260620 14:54:31.760161 30494 leader_election.cc:288] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:31.760321 30499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:31.761102 30499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 LEADER]: Becoming Leader. State: Replica: 1e4b4bf55ed040ae8799270c5b3ed33d, State: Running, Role: LEADER
I20260620 14:54:31.761759 30499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:31.762573 30494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:31.765136 30501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e4b4bf55ed040ae8799270c5b3ed33d. Latest consensus state: current_term: 1 leader_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } } }
I20260620 14:54:31.765605 30501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:31.765453 30500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } } }
I20260620 14:54:31.765874 30500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:31.772037 30505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:31.778829 30505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:31.789620 30505 catalog_manager.cc:1347] Generated new cluster ID: 01e2a6ca0ed048b78902b16026727dc7
I20260620 14:54:31.789821 30505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:31.821864 30505 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:31.823514 30505 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:31.837404 30505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Generated new TSK 0
I20260620 14:54:31.838371 30505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:54:31.903262 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38017
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:32.275151 30520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:32.275470 30520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:32.275650 30520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:32.283113 30520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:32.283295 30520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:32.283492 30520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:32.293717 30520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:32.295256 30520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:32.296722 30520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:32.311169 30525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:32.311169 30528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:32.311111 30526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:32.312250 30520 server_base.cc:1034] running on GCE node
I20260620 14:54:32.312986 30520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:32.313992 30520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:32.315166 30520 hybrid_clock.cc:648] HybridClock initialized: now 1781967272315087 us; error 66 us; skew 500 ppm
I20260620 14:54:32.315515 30520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:32 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967272, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:32.318917 30520 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:54:32.320647 30520 webserver.cc:460] Webserver started at http://127.21.159.193:43873/ using document root <none> and password file <none>
I20260620 14:54:32.321419 30520 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:32.321576 30520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:32.321903 30520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:32.325027 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "1384a12d853941779ada6ca9f6039ad9"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "e21b04d09b2fc25754bc99ee24b5f35d"
server_key_iv: "73d8119cfa050e5a238d90a8a9a81519"
server_key_version: "encryptionkey@0"
I20260620 14:54:32.325878 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "1384a12d853941779ada6ca9f6039ad9"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "e21b04d09b2fc25754bc99ee24b5f35d"
server_key_iv: "73d8119cfa050e5a238d90a8a9a81519"
server_key_version: "encryptionkey@0"
I20260620 14:54:32.331827 30520 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 14:54:32.335757 30535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:32.337365 30520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260620 14:54:32.337589 30520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "1384a12d853941779ada6ca9f6039ad9"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "e21b04d09b2fc25754bc99ee24b5f35d"
server_key_iv: "73d8119cfa050e5a238d90a8a9a81519"
server_key_version: "encryptionkey@0"
I20260620 14:54:32.337807 30520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:32.354347 30520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:32.358942 30520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:32.359313 30520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:32.360359 30520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:32.361759 30520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:32.361868 30520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:32.362018 30520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:32.362095 30520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:32.402858 30520 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:42923
I20260620 14:54:32.402874 30647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:42923 every 8 connection(s)
I20260620 14:54:32.404865 30520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:32.411466 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 30520
I20260620 14:54:32.411693 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:32.412379 22143 external_mini_cluster.cc:1428] Setting key c8312efab105e87d7e96b3c40e9fd977
Jun 20 14:54:32 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967272, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:32.427114 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:32.407661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:50805 (local address 127.21.159.254:38017)
0620 14:54:32.408127 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:32.408134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:32.409165 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 14:54:32.410981 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:32.410989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:32.411109 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:32.411248 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:32.412945 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.413826 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.415586 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.416349 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.416501 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:32.420327 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:32.420337 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:32.420345 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:32.420394 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:32.423463 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:32.424506 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:32.424511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:32.424513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:32.424651 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:32.425208 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:32.425212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:32.425214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:32.425503 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:32.425619 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:32.426595 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 14:54:32.426794 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":222,"threads_started":1}
I20260620 14:54:32.428754 30651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:32.408265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:50805)
0620 14:54:32.409001 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:32.409041 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:32.410681 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:32.411451 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:32.411460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:32.412101 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:32.412754 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:32.412774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.414051 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.414059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:32.415440 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:32.415452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.416562 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.416566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:32.416780 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:32.417508 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:32.417533 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:32.420047 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:32.423628 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:32.423641 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:32.423664 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:32.424340 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:32.424781 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:32.424788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:32.424795 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:32.425082 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:32.425637 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:32.425649 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:32.426139 (+ 490us) client_negotiation.cc:769] Sending connection context
0620 14:54:32.426495 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 14:54:32.427156 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":288,"threads_started":1}
I20260620 14:54:32.430611 30648 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38017
I20260620 14:54:32.430955 30648 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:32.431759 30648 heartbeater.cc:511] Master 127.21.159.254:38017 requested a full tablet report, sending...
I20260620 14:54:32.433956 30459 ts_manager.cc:194] Registered new tserver with Master: 1384a12d853941779ada6ca9f6039ad9 (127.21.159.193:42923)
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:54:32.435545 30459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:50805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:54:32.467022 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38017
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:32.807483 30654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:32.807853 30654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:32.808058 30654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:32.815517 30654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:32.815697 30654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:32.815884 30654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:32.825856 30654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:32.827289 30654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:32.828745 30654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:32.843318 30659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:32.843376 30662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:32.843302 30660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:32.844478 30654 server_base.cc:1034] running on GCE node
I20260620 14:54:32.845165 30654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:32.846201 30654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:32.847442 30654 hybrid_clock.cc:648] HybridClock initialized: now 1781967272847394 us; error 32 us; skew 500 ppm
I20260620 14:54:32.847829 30654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:32 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967272, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:32.851573 30654 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:54:32.853410 30654 webserver.cc:460] Webserver started at http://127.21.159.194:32839/ using document root <none> and password file <none>
I20260620 14:54:32.854212 30654 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:32.854364 30654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:32.854729 30654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:32.858255 30654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "0980242f7be341c28fc74737191ff517"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "739914b867d2ae20b4686d337b7a9f90"
server_key_iv: "14e41b7e34e7d6809ecf8bfe7aa85ca1"
server_key_version: "encryptionkey@0"
I20260620 14:54:32.859292 30654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "0980242f7be341c28fc74737191ff517"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "739914b867d2ae20b4686d337b7a9f90"
server_key_iv: "14e41b7e34e7d6809ecf8bfe7aa85ca1"
server_key_version: "encryptionkey@0"
I20260620 14:54:32.865574 30654 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:54:32.870028 30669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:32.871847 30654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 14:54:32.872077 30654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "0980242f7be341c28fc74737191ff517"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "739914b867d2ae20b4686d337b7a9f90"
server_key_iv: "14e41b7e34e7d6809ecf8bfe7aa85ca1"
server_key_version: "encryptionkey@0"
I20260620 14:54:32.872318 30654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:32.888096 30654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:32.893446 30654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:32.893818 30654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:32.894943 30654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:32.896589 30654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:32.896766 30654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 14:54:32.896914 30654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:32.896966 30654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:32.935983 30654 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:34449
I20260620 14:54:32.936026 30781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:34449 every 8 connection(s)
I20260620 14:54:32.937502 30654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:32.941526 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 30654
I20260620 14:54:32.941735 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:32.942301 22143 external_mini_cluster.cc:1428] Setting key 59b33e924df8840a9e4247195150b5ba
Jun 20 14:54:32 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967272, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:32.962925 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:32.940420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:40375 (local address 127.21.159.254:38017)
0620 14:54:32.940886 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:32.940892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:32.942293 (+ 1401us) server_negotiation.cc:407] Connection header received
0620 14:54:32.944883 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:32.944889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:32.944991 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:32.945121 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:32.946857 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.947717 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.949523 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.950105 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.950278 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:32.954812 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:32.954822 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:32.954827 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:32.954880 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:32.959004 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:32.960060 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:32.960070 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:32.960078 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:32.960269 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:32.960789 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:32.960798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:32.960813 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:32.961251 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:32.961423 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:32.962193 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 14:54:32.962476 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
I20260620 14:54:32.964202 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:32.941040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:40375)
0620 14:54:32.942091 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:32.942151 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:32.944578 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:32.945415 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:32.945425 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:32.946012 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:32.946655 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:32.946673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.947900 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.947906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:32.949344 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:32.949364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:32.950394 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:32.950403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:32.950771 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:32.951719 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:32.951746 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:32.954480 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:32.959319 (+ 4839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:32.959329 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:32.959346 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:32.959889 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:32.960430 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:32.960434 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:32.960437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:32.960648 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:32.961398 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:32.961405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:32.961782 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 14:54:32.962104 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 14:54:32.962803 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":479,"threads_started":1}
I20260620 14:54:32.965879 30782 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38017
I20260620 14:54:32.966282 30782 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:54:32.967454 30782 heartbeater.cc:511] Master 127.21.159.254:38017 requested a full tablet report, sending...
I20260620 14:54:32.969129 30459 ts_manager.cc:194] Registered new tserver with Master: 0980242f7be341c28fc74737191ff517 (127.21.159.194:34449)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:54:32.970180 30459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:40375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:54:32.998917 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38017
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:33.353052 30788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:33.353382 30788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:33.353490 30788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:33.362010 30788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:33.362149 30788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:33.362275 30788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:33.374018 30788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:33.375578 30788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:33.377127 30788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:33.392544 30793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:33.392627 30796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:33.392591 30794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:33.393570 30788 server_base.cc:1034] running on GCE node
I20260620 14:54:33.394275 30788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:33.395504 30788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:33.396809 30788 hybrid_clock.cc:648] HybridClock initialized: now 1781967273396753 us; error 70 us; skew 500 ppm
I20260620 14:54:33.397198 30788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:33 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:33.401187 30788 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:54:33.407429 30788 webserver.cc:460] Webserver started at http://127.21.159.195:38855/ using document root <none> and password file <none>
I20260620 14:54:33.408221 30788 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:33.408341 30788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:33.408720 30788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:33.411873 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "4e3c1a48394c4da8914e33e2a91c40c0"
format_stamp: "Formatted at 2026-06-20 14:54:33 on dist-test-slave-3mqx"
server_key: "1e934db5457dae4f10d39d124d43e0ed"
server_key_iv: "853d0b789544bfe42719912bbc930eab"
server_key_version: "encryptionkey@0"
I20260620 14:54:33.412711 30788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "4e3c1a48394c4da8914e33e2a91c40c0"
format_stamp: "Formatted at 2026-06-20 14:54:33 on dist-test-slave-3mqx"
server_key: "1e934db5457dae4f10d39d124d43e0ed"
server_key_iv: "853d0b789544bfe42719912bbc930eab"
server_key_version: "encryptionkey@0"
I20260620 14:54:33.418605 30788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:54:33.422814 30803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:33.424384 30788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:33.424652 30788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "4e3c1a48394c4da8914e33e2a91c40c0"
format_stamp: "Formatted at 2026-06-20 14:54:33 on dist-test-slave-3mqx"
server_key: "1e934db5457dae4f10d39d124d43e0ed"
server_key_iv: "853d0b789544bfe42719912bbc930eab"
server_key_version: "encryptionkey@0"
I20260620 14:54:33.424880 30788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:33.439110 30648 heartbeater.cc:503] Master 127.21.159.254:38017 was elected leader, sending a full tablet report...
I20260620 14:54:33.458770 30788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:33.463989 30788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:33.464295 30788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:33.465344 30788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:33.466766 30788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:33.466871 30788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:33.467017 30788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:33.467121 30788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 14:54:33.514482 30788 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:44309
I20260620 14:54:33.514590 30915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:44309 every 8 connection(s)
I20260620 14:54:33.516265 30788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:33.517879 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 30788
I20260620 14:54:33.518071 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:33.518582 22143 external_mini_cluster.cc:1428] Setting key 34b9679f6f5784653af9b7386769cac7
Jun 20 14:54:33 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967273, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:33.542495 30919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:33.520960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:44415 (local address 127.21.159.254:38017)
0620 14:54:33.522328 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:33.522332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:33.522349 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:54:33.524176 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:33.524180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:33.524260 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:33.524353 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:33.526859 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.527778 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.530910 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.531434 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.531542 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:33.536136 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:33.536142 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:33.536147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:33.536191 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:33.538964 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.539787 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.539795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.539801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.539927 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.540392 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.540398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.540402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.540683 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:33.540845 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:33.541753 (+ 908us) server_negotiation.cc:299] Negotiation successful
0620 14:54:33.542001 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1271,"thread_start_us":1148,"threads_started":1}
I20260620 14:54:33.543607 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:33.520339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:44415)
0620 14:54:33.521295 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:33.521362 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:33.523832 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:33.524717 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:33.524732 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:33.525645 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:33.526618 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:33.526652 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.527971 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.527979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:33.530704 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:33.530726 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.531717 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.531726 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:33.532135 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:33.533073 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:33.533107 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:33.535907 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:33.539152 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:33.539162 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:33.539182 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:33.539653 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:33.540058 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:33.540063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:33.540067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:33.540274 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:33.540827 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:33.540836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:33.541339 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 14:54:33.541671 (+ 332us) client_negotiation.cc:240] Negotiation successful
0620 14:54:33.542255 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":388,"threads_started":1}
I20260620 14:54:33.545248 30916 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38017
I20260620 14:54:33.545639 30916 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:33.546685 30916 heartbeater.cc:511] Master 127.21.159.254:38017 requested a full tablet report, sending...
I20260620 14:54:33.548805 30459 ts_manager.cc:194] Registered new tserver with Master: 4e3c1a48394c4da8914e33e2a91c40c0 (127.21.159.195:44309)
I20260620 14:54:33.550247 30459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:44415
I20260620 14:54:33.560204 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab.
Entry for principal oryx/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab.
WARNING: no policy specified for oryx/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab.
Entry for principal oryx/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab.
WARNING: no policy specified for oryx/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab.
Entry for principal oryx/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab.
I20260620 14:54:33.712399 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 30520
Jun 20 14:54:33 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:54:33.732242 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:33.713731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.21.159.193:42923)
0620 14:54:33.714373 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:33.714380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:33.714408 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:54:33.714468 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:33.714477 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:33.714722 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:33.714855 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:33.716650 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.717663 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.719518 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.720103 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.720219 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:33.725218 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:33.725233 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:33.725259 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:33.725298 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:33.728762 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.729820 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.729830 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.729832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.729978 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.730514 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.730519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.730521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.730884 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:33.731216 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:33.731682 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 14:54:33.731909 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":327,"threads_started":1}
I20260620 14:54:33.733645 30643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:33.818311 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 30520
I20260620 14:54:33.837338 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 30654
Jun 20 14:54:33 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:54:33.855367 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:33.838287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.21.159.194:34449)
0620 14:54:33.838870 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:33.838890 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:33.838916 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:54:33.838978 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:33.838984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:33.839287 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:33.839441 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:33.840994 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.842180 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.843643 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.844229 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.844346 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:33.848883 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:33.848897 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:33.848914 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:33.848955 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:33.852093 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.852895 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.852902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.852905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.853051 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.853425 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.853430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.853432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.853820 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:33.854105 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:33.854449 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:54:33.854755 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":343,"threads_started":1}
I20260620 14:54:33.856734 30777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:33.940564 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 30654
I20260620 14:54:33.955432 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 30788
Jun 20 14:54:33 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:54:33.975279 30931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:33.956750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.21.159.195:44309)
0620 14:54:33.957316 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:33.957324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:33.957357 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:54:33.957507 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:33.957514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:33.957731 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:33.957879 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:33.959503 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.960556 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.962095 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:33.962894 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:33.963030 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:33.968390 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:33.968407 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:33.968424 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:33.968470 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:33.971882 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.972809 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.972818 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.972821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.973053 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:33.973471 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:33.973475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:33.973478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:33.974038 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:33.974347 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:33.974677 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:54:33.974894 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":276,"threads_started":1}
I20260620 14:54:33.976686 30911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:34.061959 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 30788
I20260620 14:54:34.078328 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 30427
I20260620 14:54:34.079708 30489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:34.181777 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 30427
I20260620 14:54:34.203012 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38017
--webserver_interface=127.21.159.254
--webserver_port=33763
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:34.547739 30934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:34.548055 30934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:34.548216 30934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:34.555896 30934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:34.556013 30934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:34.556100 30934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:34.556188 30934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:34.556304 30934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:34.566466 30934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=33763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:34.567987 30934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:34.569428 30934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:34.581813 30942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:34.581907 30940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:34.582484 30939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:34.583302 30934 server_base.cc:1034] running on GCE node
I20260620 14:54:34.584177 30934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:34.586373 30934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:34.587934 30934 hybrid_clock.cc:648] HybridClock initialized: now 1781967274587856 us; error 60 us; skew 500 ppm
I20260620 14:54:34.588441 30934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:34 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967274, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:34.593501 30934 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:54:34.596375 30934 webserver.cc:460] Webserver started at http://127.21.159.254:33763/ using document root <none> and password file <none>
I20260620 14:54:34.597282 30934 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:34.597409 30934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:34.603852 30934 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:54:34.607210 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:34.608731 30934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:54:34.608947 30934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "1e4b4bf55ed040ae8799270c5b3ed33d"
format_stamp: "Formatted at 2026-06-20 14:54:31 on dist-test-slave-3mqx"
server_key: "42b3ce1ba630c673e17f7bce28894e72"
server_key_iv: "0058d5f4c4ec2c03098eb42b4a94bae0"
server_key_version: "encryptionkey@0"
I20260620 14:54:34.610044 30934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:34.636236 30934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:34.641000 30934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:34.641291 30934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:34.661257 30934 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:38017
I20260620 14:54:34.661274 31000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:38017 every 8 connection(s)
I20260620 14:54:34.662863 30934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:34.666632 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 30934
I20260620 14:54:34.667526 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:42923
--local_ip_for_outbound_sockets=127.21.159.193
--tserver_master_addrs=127.21.159.254:38017
--webserver_port=43873
--webserver_interface=127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:54:34.670218 31001 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:54:34.674007 31001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Bootstrap starting.
I20260620 14:54:34.703429 31001 log.cc:824] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:34.715492 31001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:54:34.716171 31001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Bootstrap complete.
I20260620 14:54:34.726758 31001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:34.727823 31001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4b4bf55ed040ae8799270c5b3ed33d, State: Initialized, Role: FOLLOWER
I20260620 14:54:34.728559 31001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:34.728825 31001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:34.728960 31001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:34.729151 31001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:54:34.733218 31001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:34.733789 31001 leader_election.cc:302] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e4b4bf55ed040ae8799270c5b3ed33d; no voters:
I20260620 14:54:34.734486 31001 leader_election.cc:288] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:54:34.734738 31005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:54:34.735872 31005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [term 2 LEADER]: Becoming Leader. State: Replica: 1e4b4bf55ed040ae8799270c5b3ed33d, State: Running, Role: LEADER
I20260620 14:54:34.736601 31005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } }
I20260620 14:54:34.736840 31001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:34.742938 31006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } } }
I20260620 14:54:34.743448 31006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:34.744105 31007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e4b4bf55ed040ae8799270c5b3ed33d. Latest consensus state: current_term: 2 leader_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e4b4bf55ed040ae8799270c5b3ed33d" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38017 } } }
I20260620 14:54:34.744470 31007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:34.751573 31015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:34.758105 31015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:34.760118 31015 catalog_manager.cc:1259] Loaded cluster ID: 01e2a6ca0ed048b78902b16026727dc7
I20260620 14:54:34.760385 31015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:34.764849 31023 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:54:34.765730 31015 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:34.767102 31015 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1e4b4bf55ed040ae8799270c5b3ed33d: Loaded TSK: 0
I20260620 14:54:34.768528 31015 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:54:35.067853 31003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:35.068162 31003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:35.068295 31003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:35.075950 31003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:35.076071 31003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:35.076244 31003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:35.086519 31003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.193.keytab
--principal=oryx/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:42923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=43873
--tserver_master_addrs=127.21.159.254:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:35.088050 31003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:35.089464 31003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:35.104883 31029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:35.105525 31028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:35.108263 31031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:35.108968 31003 server_base.cc:1034] running on GCE node
I20260620 14:54:35.109635 31003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:35.110721 31003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:35.111977 31003 hybrid_clock.cc:648] HybridClock initialized: now 1781967275111918 us; error 55 us; skew 500 ppm
I20260620 14:54:35.112320 31003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967275, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:35.115928 31003 init.cc:377] Logged in from keytab as oryx/127.21.159.193@KRBTEST.COM (short username oryx)
I20260620 14:54:35.117715 31003 webserver.cc:460] Webserver started at http://127.21.159.193:43873/ using document root <none> and password file <none>
I20260620 14:54:35.118479 31003 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:35.118623 31003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:35.124660 31003 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:54:35.127766 31038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:35.129340 31003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:54:35.129528 31003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "1384a12d853941779ada6ca9f6039ad9"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "e21b04d09b2fc25754bc99ee24b5f35d"
server_key_iv: "73d8119cfa050e5a238d90a8a9a81519"
server_key_version: "encryptionkey@0"
I20260620 14:54:35.130532 31003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:35.149462 31003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:35.154352 31003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:35.154651 31003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:35.155879 31003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:35.157410 31003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:35.157500 31003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:35.157652 31003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:35.157732 31003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:35.193460 31003 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:42923
I20260620 14:54:35.193558 31150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:42923 every 8 connection(s)
I20260620 14:54:35.195120 31003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:35.196727 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31003
I20260620 14:54:35.197532 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:34449
--local_ip_for_outbound_sockets=127.21.159.194
--tserver_master_addrs=127.21.159.254:38017
--webserver_port=32839
--webserver_interface=127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:35.217061 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:35.198976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:58763)
0620 14:54:35.199939 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:35.199999 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:35.202764 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:35.204778 (+ 2014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:35.204792 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:35.205449 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:35.206038 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.206057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.207799 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.207803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.209143 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.209157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.210258 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.210265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.210571 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.211413 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:35.211438 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:35.215589 (+ 4151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":348,"threads_started":1}
W20260620 14:54:35.217087 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:35.198986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:58763 (local address 127.21.159.254:38017)
0620 14:54:35.203371 (+ 4385us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:35.203385 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:35.203430 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:54:35.203766 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:35.203782 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:35.204189 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:35.204605 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:35.206282 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.207599 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.209312 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.209988 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.210228 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.215986 (+ 5758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:58763: BlockingRecv error: recv got EOF from 127.21.159.193:58763 (error 108)
Metrics: {"server-negotiator.queue_time_us":4096,"thread_start_us":3821,"threads_started":1}
W20260620 14:54:35.217796 31151 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:38017 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.159.254:38017: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:35.227890 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:35.218514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:42547)
0620 14:54:35.218737 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:35.218760 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:35.218876 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:35.219344 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:35.219347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:35.219674 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:35.219897 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.219910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.221578 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.221582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.222510 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.222522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.223513 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.223518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.223646 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.224334 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:35.224353 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:35.227570 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 14:54:35.228006 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:35.218707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:42547 (local address 127.21.159.254:38017)
0620 14:54:35.218925 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:35.218933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:35.218957 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:54:35.219042 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:35.219050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:35.219160 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:35.219278 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:35.220094 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.221412 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.222647 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.223354 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.223479 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.227795 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:42547: BlockingRecv error: recv got EOF from 127.21.159.193:42547 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:35.237903 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:35.228636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:57849)
0620 14:54:35.228832 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:35.228854 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:35.228982 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:35.229347 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:35.229351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:35.229637 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:35.229783 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.229791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.231204 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.231208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.232116 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.232128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.233181 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.233186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.233456 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.234197 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:35.234228 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:35.237654 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 14:54:35.238152 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:35.228843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:57849 (local address 127.21.159.254:38017)
0620 14:54:35.229033 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:35.229040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:35.229059 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:54:35.229117 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:35.229123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:35.229209 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:35.229322 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:35.229914 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.231032 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.232342 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.233001 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.233139 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.237898 (+ 4759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:57849: BlockingRecv error: recv got EOF from 127.21.159.193:57849 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 14:54:35.239257 31151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:54:35.553073 31155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:35.553408 31155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:35.553575 31155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:35.561153 31155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:35.561281 31155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:35.561482 31155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:35.571622 31155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.194.keytab
--principal=oryx/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:34449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=32839
--tserver_master_addrs=127.21.159.254:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:35.573215 31155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:35.574654 31155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:35.589265 31163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:35.589341 31161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:35.589229 31160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:35.595881 31155 server_base.cc:1034] running on GCE node
I20260620 14:54:35.596632 31155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:35.597812 31155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:35.599180 31155 hybrid_clock.cc:648] HybridClock initialized: now 1781967275598998 us; error 66 us; skew 500 ppm
I20260620 14:54:35.599567 31155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967275, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:35.603534 31155 init.cc:377] Logged in from keytab as oryx/127.21.159.194@KRBTEST.COM (short username oryx)
I20260620 14:54:35.605444 31155 webserver.cc:460] Webserver started at http://127.21.159.194:32839/ using document root <none> and password file <none>
I20260620 14:54:35.606256 31155 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:35.606384 31155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:35.613008 31155 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:54:35.616448 31170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:35.618144 31155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 14:54:35.618472 31155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "0980242f7be341c28fc74737191ff517"
format_stamp: "Formatted at 2026-06-20 14:54:32 on dist-test-slave-3mqx"
server_key: "739914b867d2ae20b4686d337b7a9f90"
server_key_iv: "14e41b7e34e7d6809ecf8bfe7aa85ca1"
server_key_version: "encryptionkey@0"
I20260620 14:54:35.619666 31155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:35.642048 31155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:35.646881 31155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:35.647269 31155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:35.648377 31155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:35.649864 31155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:35.649991 31155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:35.650134 31155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:35.650223 31155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:35.687317 31155 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:34449
I20260620 14:54:35.687377 31282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:34449 every 8 connection(s)
I20260620 14:54:35.688840 31155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:35.693444 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31155
I20260620 14:54:35.694154 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:44309
--local_ip_for_outbound_sockets=127.21.159.195
--tserver_master_addrs=127.21.159.254:38017
--webserver_port=38855
--webserver_interface=127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:54:35.706499 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:35.691951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:39377 (local address 127.21.159.254:38017)
0620 14:54:35.692169 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:35.692176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:35.693462 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 14:54:35.695753 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:35.695759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:35.695854 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:35.695970 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:35.697608 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.698349 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.700071 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.700517 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.700660 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.706241 (+ 5581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:39377: BlockingRecv error: recv got EOF from 127.21.159.194:39377 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:54:35.707378 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:35.692478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:39377)
0620 14:54:35.693261 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:35.693301 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:35.695419 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:35.696251 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:35.696260 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:35.696793 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:35.697408 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.697427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.698522 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.698534 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.699916 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.699928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.700773 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.700780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.701017 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.701865 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:35.701893 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:35.706013 (+ 4120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":330,"threads_started":1}
W20260620 14:54:35.708148 31283 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:38017 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.159.254:38017: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:35.720503 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:35.708872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:38709)
0620 14:54:35.709195 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:35.709220 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:35.709360 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:35.709785 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:35.709793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:35.710336 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:35.710586 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.710602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.711944 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.711952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.713985 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.714008 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.715120 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.715130 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.715387 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.716147 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:35.716182 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:35.720134 (+ 3952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158}
W20260620 14:54:35.720700 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:35.709018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:38709 (local address 127.21.159.254:38017)
0620 14:54:35.709203 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:35.709208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:35.709301 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:54:35.709504 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:35.709508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:35.709588 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:35.709748 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:35.710807 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.711781 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.714229 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.714904 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.715022 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.720391 (+ 5369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:38709: BlockingRecv error: recv got EOF from 127.21.159.194:38709 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:35 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:35.730360 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:35.721470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:53029)
0620 14:54:35.721673 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:35.721751 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:35.721868 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:35.722219 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:35.722222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:35.722505 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:35.722650 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.722658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.724295 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.724299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.725320 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:35.725332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.726467 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.726472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:35.726600 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.727303 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:35.727321 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:35.730135 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 14:54:35.730749 31283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:54:35.730790 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:35.721695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:53029 (local address 127.21.159.254:38017)
0620 14:54:35.721887 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:35.721894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:35.721943 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:54:35.722005 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:35.722011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:35.722090 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:35.722190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:35.722814 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.724124 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.725463 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:35.726302 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:35.726452 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:35.730427 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:53029: BlockingRecv error: recv got EOF from 127.21.159.194:53029 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:54:36.044011 31286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:36.044319 31286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:36.044417 31286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:36.051996 31286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:36.052122 31286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:36.052248 31286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:36.062222 31286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/krb5kdc/oryx_127.21.159.195.keytab
--principal=oryx/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:44309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=38855
--tserver_master_addrs=127.21.159.254:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:36.063581 31286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:36.064862 31286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:36.079020 31294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:36.078996 31292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:36.079696 31291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:36.079941 31286 server_base.cc:1034] running on GCE node
I20260620 14:54:36.080627 31286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:36.081672 31286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:36.082942 31286 hybrid_clock.cc:648] HybridClock initialized: now 1781967276082887 us; error 47 us; skew 500 ppm
I20260620 14:54:36.083333 31286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967276, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:36.086689 31286 init.cc:377] Logged in from keytab as oryx/127.21.159.195@KRBTEST.COM (short username oryx)
I20260620 14:54:36.088479 31286 webserver.cc:460] Webserver started at http://127.21.159.195:38855/ using document root <none> and password file <none>
I20260620 14:54:36.089265 31286 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:36.089444 31286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:36.095479 31286 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:54:36.098536 31301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:36.100376 31286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:36.100587 31286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "4e3c1a48394c4da8914e33e2a91c40c0"
format_stamp: "Formatted at 2026-06-20 14:54:33 on dist-test-slave-3mqx"
server_key: "1e934db5457dae4f10d39d124d43e0ed"
server_key_iv: "853d0b789544bfe42719912bbc930eab"
server_key_version: "encryptionkey@0"
I20260620 14:54:36.101634 31286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:36.126688 31286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:36.131731 31286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:36.132058 31286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:36.133193 31286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:36.134882 31286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:36.135025 31286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:36.135193 31286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:36.135270 31286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:36.172588 31286 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:44309
I20260620 14:54:36.172649 31413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:44309 every 8 connection(s)
I20260620 14:54:36.174202 31286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:36.180500 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31286
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:54:36.194213 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:36.177063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:55105 (local address 127.21.159.254:38017)
0620 14:54:36.177248 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:36.177252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:36.178431 (+ 1179us) server_negotiation.cc:407] Connection header received
0620 14:54:36.180480 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:36.180486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:36.180616 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:36.180743 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:36.183012 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.184356 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.186734 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.187506 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.187654 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.193910 (+ 6256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:55105: BlockingRecv error: recv got EOF from 127.21.159.195:55105 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":58}
I20260620 14:54:36.195158 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:36.177508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:55105)
0620 14:54:36.178250 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:36.178300 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:36.180184 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:36.181028 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:36.181037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:36.181855 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:36.182761 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.182798 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.184530 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.184535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.186563 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.186577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.187764 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.187770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.188056 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.188890 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:36.188936 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:36.193698 (+ 4762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":317,"threads_started":1}
W20260620 14:54:36.198362 31414 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:38017 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.159.254:38017: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
I20260620 14:54:36.201299 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:36.181573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.21.159.254:38017)
0620 14:54:36.182715 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:36.182722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:36.182740 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:54:36.182794 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:36.182799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:36.182873 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:36.182979 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:36.184975 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.186153 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.188179 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.188848 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.188981 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.192460 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:36.192477 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:36.192504 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:36.192560 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:36.196980 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:36.198083 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:36.198092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:36.198101 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:36.198369 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:36.198895 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:36.198899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:36.198901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:36.199651 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:36.200007 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:36.200502 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:54:36.200845 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":1016,"thread_start_us":871,"threads_started":1}
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:36.208602 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:36.199091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:37671)
0620 14:54:36.199376 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:36.199396 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:36.199522 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:36.199972 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:36.199977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:36.200449 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:36.200642 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.200655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.202077 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.202083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.203442 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.203457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.204414 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.204418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.204547 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.205261 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:36.205297 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:36.208395 (+ 3098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163}
W20260620 14:54:36.208925 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:36.199195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:37671 (local address 127.21.159.254:38017)
0620 14:54:36.199365 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:36.199370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:36.199475 (+ 105us) server_negotiation.cc:407] Connection header received
0620 14:54:36.199641 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:36.199646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:36.199728 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:36.199933 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:36.200865 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.201924 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.203589 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.204244 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.204348 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.208600 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:37671: BlockingRecv error: recv got EOF from 127.21.159.195:37671 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:36.218551 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:36.209560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:39875)
0620 14:54:36.209750 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:36.209771 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:36.209920 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:36.210206 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:36.210209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:36.210487 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:36.210633 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.210641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.211907 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.211912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.212886 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.212898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.213712 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.213717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.214031 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.214849 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:36.214867 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:36.218220 (+ 3353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:54:36.218674 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:36.209648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:39875 (local address 127.21.159.254:38017)
0620 14:54:36.209936 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:36.209940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:36.209977 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:54:36.210024 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:36.210028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:36.210095 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:36.210169 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:36.210797 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.211669 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.213083 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.213538 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.213649 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.218425 (+ 4776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:39875: BlockingRecv error: recv got EOF from 127.21.159.195:39875 (error 108)
Metrics: {"server-negotiator.queue_time_us":195}
W20260620 14:54:36.218987 31414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:36.250674 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:36.239877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:55517)
0620 14:54:36.240479 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:36.240498 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:36.240638 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:36.241017 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:36.241022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:36.241694 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:36.241971 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.241986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.243535 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.243545 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.245116 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.245132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.246257 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.246265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.246440 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.247161 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:36.247182 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:36.250384 (+ 3202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":316,"threads_started":1}
W20260620 14:54:36.250948 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:36.240093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:55517 (local address 127.21.159.254:38017)
0620 14:54:36.240323 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:36.240330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:36.240590 (+ 260us) server_negotiation.cc:407] Connection header received
0620 14:54:36.240749 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:36.240754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:36.240849 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:36.240938 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:36.242186 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.243354 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.245286 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.245986 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.246139 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.250663 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:55517: BlockingRecv error: recv got EOF from 127.21.159.193:55517 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:36 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:36.745995 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:36.731406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:51401)
0620 14:54:36.731918 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:36.731934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:36.732105 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:36.732467 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:36.732472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:36.732976 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:36.733183 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.733192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.734267 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.734273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.735359 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:36.735372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.736161 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.736167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:36.736349 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.742164 (+ 5815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:36.742188 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:36.745623 (+ 3435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":268,"threads_started":1}
W20260620 14:54:36.746179 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:36.731504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:51401 (local address 127.21.159.254:38017)
0620 14:54:36.731742 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:36.731748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:36.732026 (+ 278us) server_negotiation.cc:407] Connection header received
0620 14:54:36.732249 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:36.732254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:36.732324 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:36.732432 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:36.733345 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.734111 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.735514 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:36.735965 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:36.736114 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:36.745874 (+ 9760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:51401: BlockingRecv error: recv got EOF from 127.21.159.194:51401 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 14:54:37 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:37 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:37.229009 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:37.219719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:40713)
0620 14:54:37.220264 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:37.220286 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:37.220416 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:37.220746 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:37.220750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:37.221231 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:37.221424 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:37.221433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.222535 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.222539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:37.223680 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:37.223695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.224720 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.224731 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:37.224988 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:37.225740 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:37.225763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:37.228783 (+ 3020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":321,"threads_started":1}
W20260620 14:54:37.229264 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:37.219853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:40713 (local address 127.21.159.254:38017)
0620 14:54:37.220138 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:37.220144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:37.220428 (+ 284us) server_negotiation.cc:407] Connection header received
0620 14:54:37.220522 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:37.220526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:37.220595 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:37.220701 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:37.221584 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.222384 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.223890 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.224535 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.224672 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:37.229027 (+ 4355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:40713: BlockingRecv error: recv got EOF from 127.21.159.195:40713 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Jun 20 14:54:37 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:37 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:37.259953 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:37.251665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:45847)
0620 14:54:37.252157 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:37.252174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:37.252297 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:37.252574 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:37.252577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:37.252949 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:37.253115 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:37.253124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.254107 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.254110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:37.255011 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:37.255022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.255760 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.255764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:37.255875 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:37.256446 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:37.256462 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:37.259626 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":283,"threads_started":1}
W20260620 14:54:37.260098 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:37.251735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:45847 (local address 127.21.159.254:38017)
0620 14:54:37.251998 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:37.252004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:37.252255 (+ 251us) server_negotiation.cc:407] Connection header received
0620 14:54:37.252397 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:37.252401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:37.252477 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:37.252559 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:37.253261 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.253983 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.255194 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.255626 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.255720 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:37.259846 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:45847: BlockingRecv error: recv got EOF from 127.21.159.193:45847 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:54:37 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:37 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:37.756027 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:37.747253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:48191)
0620 14:54:37.747786 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:37.747806 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:37.747949 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:37.748346 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:37.748351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:37.748994 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:37.749175 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:37.749185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.750174 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.750177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:37.751300 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:37.751319 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.752107 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.752112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:37.752221 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:37.752822 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:37.752841 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:37.755788 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":287,"threads_started":1}
W20260620 14:54:37.756286 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:37.747347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:48191 (local address 127.21.159.254:38017)
0620 14:54:37.747596 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:37.747602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:37.747906 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:54:37.748079 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:37.748085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:37.748161 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:37.748272 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:37.749308 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.750051 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.751438 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:37.751960 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:37.752061 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:37.756034 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:48191: BlockingRecv error: recv got EOF from 127.21.159.194:48191 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 14:54:38 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:38 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:38.239125 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.229965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:59369)
0620 14:54:38.230551 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.230571 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:38.230733 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:38.231124 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:38.231129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:38.231590 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:38.231758 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:38.231767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.232872 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.232879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:38.233861 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:38.233874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.234672 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.234678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:38.234848 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:38.235594 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:38.235636 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:38.238739 (+ 3103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":335,"threads_started":1}
W20260620 14:54:38.239265 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:38.230189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:59369 (local address 127.21.159.254:38017)
0620 14:54:38.230415 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:38.230421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:38.230655 (+ 234us) server_negotiation.cc:407] Connection header received
0620 14:54:38.230865 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:38.230870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:38.230942 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:38.231061 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:38.231906 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.232721 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.234015 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.234468 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.234638 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:38.238966 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:59369: BlockingRecv error: recv got EOF from 127.21.159.195:59369 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 14:54:38 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:38 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:38.270578 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.261085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:53101)
0620 14:54:38.261725 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.261742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:38.261911 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:38.262311 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:38.262315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:38.262696 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:38.262854 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:38.262863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.263964 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.263971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:38.264900 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:38.264910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.265678 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.265684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:38.265968 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:38.266867 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:38.266888 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:38.270202 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":341,"threads_started":1}
W20260620 14:54:38.270676 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:38.261324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:53101 (local address 127.21.159.254:38017)
0620 14:54:38.261604 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:38.261611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:38.261835 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:54:38.262114 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:38.262119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:38.262180 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:38.262287 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:38.263015 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.263813 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.265055 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.265502 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.265636 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:38.270420 (+ 4784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:53101: BlockingRecv error: recv got EOF from 127.21.159.193:53101 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 14:54:38 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:38 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:38.766781 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.757139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:50453)
0620 14:54:38.757675 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.757693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:38.757852 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:38.758259 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:38.758263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:38.758642 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:38.758812 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:38.758820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.760124 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.760128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:38.761249 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:38.761267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.762160 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.762165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:38.762395 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:38.763247 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:38.763271 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:38.766449 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":292,"threads_started":1}
W20260620 14:54:38.766992 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:38.757259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:50453 (local address 127.21.159.254:38017)
0620 14:54:38.757530 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:38.757536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:38.757798 (+ 262us) server_negotiation.cc:407] Connection header received
0620 14:54:38.758010 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:38.758015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:38.758078 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:38.758183 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:38.759026 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.759950 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.761464 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:38.761984 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:38.762139 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:38.766733 (+ 4594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:50453: BlockingRecv error: recv got EOF from 127.21.159.194:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 14:54:39 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:39 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:39.249955 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:39.240240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:35773)
0620 14:54:39.240752 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:39.240775 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:39.240893 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:39.241273 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:39.241278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:39.241621 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:39.241787 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:39.241796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.242850 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.242855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:39.244343 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:39.244361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.245364 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.245373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:39.245559 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:39.246361 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:39.246384 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:39.249665 (+ 3281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":300,"threads_started":1}
W20260620 14:54:39.250114 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:39.240461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:35773 (local address 127.21.159.254:38017)
0620 14:54:39.240633 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:39.240637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:39.240855 (+ 218us) server_negotiation.cc:407] Connection header received
0620 14:54:39.241009 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:39.241013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:39.241076 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:39.241179 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:39.241931 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.242632 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.244535 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.245187 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.245316 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:39.249911 (+ 4595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:35773: BlockingRecv error: recv got EOF from 127.21.159.195:35773 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:54:39 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:39 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:39.280500 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:39.271671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:56009)
0620 14:54:39.272181 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:39.272198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:39.272321 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:39.272678 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:39.272682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:39.273029 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:39.273182 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:39.273194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.274488 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.274493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:39.275538 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:39.275569 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.276372 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.276377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:39.276513 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:39.277132 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:39.277150 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:39.280254 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":285,"threads_started":1}
W20260620 14:54:39.280799 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:39.271740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:56009 (local address 127.21.159.254:38017)
0620 14:54:39.271942 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:39.271946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:39.272290 (+ 344us) server_negotiation.cc:407] Connection header received
0620 14:54:39.272457 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:39.272461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:39.272525 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:39.272612 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:39.273394 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.274323 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.275713 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.276206 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.276310 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:39.280479 (+ 4169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:56009: BlockingRecv error: recv got EOF from 127.21.159.193:56009 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 14:54:39 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:39 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:39.777061 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:39.767765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:54581)
0620 14:54:39.768351 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:39.768367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:39.768542 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:39.768923 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:39.768928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:39.769383 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:39.769530 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:39.769539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.770612 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.770620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:39.771739 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:39.771753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.772522 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.772528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:39.772710 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:39.773398 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:39.773416 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:39.776774 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":336,"threads_started":1}
W20260620 14:54:39.777406 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:39.767919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:54581 (local address 127.21.159.254:38017)
0620 14:54:39.768180 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:39.768188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:39.768463 (+ 275us) server_negotiation.cc:407] Connection header received
0620 14:54:39.768679 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:39.768685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:39.768776 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:39.768891 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:39.769687 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.770465 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.771904 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:39.772351 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:39.772501 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:39.777111 (+ 4610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:54581: BlockingRecv error: recv got EOF from 127.21.159.194:54581 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 14:54:40 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:40 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:40.260952 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.251210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:37789)
0620 14:54:40.251708 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.251732 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:40.251928 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:40.252346 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:40.252350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:40.252754 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:40.252918 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.252927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.253994 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.253998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.255226 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.255240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.256051 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.256057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.256237 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.256914 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:40.256939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:40.260552 (+ 3613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
W20260620 14:54:40.261134 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:40.251235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:37789 (local address 127.21.159.254:38017)
0620 14:54:40.251471 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:40.251481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:40.251828 (+ 347us) server_negotiation.cc:407] Connection header received
0620 14:54:40.252075 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:40.252079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:40.252148 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:40.252236 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:40.253068 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.253853 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.255430 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.255854 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.256035 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.260834 (+ 4799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:37789: BlockingRecv error: recv got EOF from 127.21.159.195:37789 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:54:40 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:40 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:40.290836 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.281641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:51679)
0620 14:54:40.282330 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.282356 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:40.282531 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:40.282892 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:40.282895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:40.283429 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:40.283601 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.283611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.284708 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.284712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.285700 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.285711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.286500 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.286504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.286647 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.287452 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:40.287472 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:40.290559 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":393,"threads_started":1}
W20260620 14:54:40.291213 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:40.281802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:51679 (local address 127.21.159.254:38017)
0620 14:54:40.282066 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:40.282072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:40.282452 (+ 380us) server_negotiation.cc:407] Connection header received
0620 14:54:40.282680 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:40.282685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:40.282755 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:40.282839 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:40.283758 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.284571 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.285855 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.286300 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.286464 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.290894 (+ 4430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:51679: BlockingRecv error: recv got EOF from 127.21.159.193:51679 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 14:54:40 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:40 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:40.787847 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.778372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:45817)
0620 14:54:40.778955 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.778976 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:40.779138 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:40.779531 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:40.779538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:40.779972 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:40.780135 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.780145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.781245 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.781249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.782206 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.782218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.783030 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.783034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.783206 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.783828 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:40.783847 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:40.787469 (+ 3622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":273,"threads_started":1}
W20260620 14:54:40.787952 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:40.778597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:45817 (local address 127.21.159.254:38017)
0620 14:54:40.778812 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:40.778817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:40.779132 (+ 315us) server_negotiation.cc:407] Connection header received
0620 14:54:40.779251 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:40.779255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:40.779321 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:40.779479 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:40.780321 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.781096 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.782373 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.782847 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.782957 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.787692 (+ 4735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:45817: BlockingRecv error: recv got EOF from 127.21.159.194:45817 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 14:54:41 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:41 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:41.272080 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:41.262562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:39613)
0620 14:54:41.263357 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:41.263385 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:41.263539 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:41.264116 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:41.264122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:41.264712 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:41.264929 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:41.264942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.266468 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.266472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:41.267506 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:41.267517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.268173 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.268177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:41.268315 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:41.268936 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:41.268954 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:41.271821 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":291,"threads_started":1}
W20260620 14:54:41.272387 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:41.262983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:39613 (local address 127.21.159.254:38017)
0620 14:54:41.263310 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:41.263316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:41.263506 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:54:41.263727 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:41.263735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:41.263823 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:41.263932 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:41.265110 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.266324 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.267646 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.268052 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.268147 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:41.272075 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:39613: BlockingRecv error: recv got EOF from 127.21.159.195:39613 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Jun 20 14:54:41 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:41 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:41.300639 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:41.291795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:46575)
0620 14:54:41.292385 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:41.292403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:41.292523 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:41.292828 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:41.292831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:41.293222 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:41.293400 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:41.293410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.294451 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.294455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:41.295701 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:41.295715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.296427 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.296432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:41.296568 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:41.297193 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:41.297212 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:41.300341 (+ 3129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":334,"threads_started":1}
W20260620 14:54:41.300899 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:41.291880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:46575 (local address 127.21.159.254:38017)
0620 14:54:41.292058 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:41.292062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:41.292477 (+ 415us) server_negotiation.cc:407] Connection header received
0620 14:54:41.292624 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:41.292627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:41.292688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:41.292773 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:41.293527 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.294296 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.295845 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.296269 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.296375 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:41.300597 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:46575: BlockingRecv error: recv got EOF from 127.21.159.193:46575 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:54:41 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:41 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:41.804476 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:41.789038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:56579)
0620 14:54:41.789748 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:41.789776 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:41.789915 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:41.790296 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:41.790301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:41.790738 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:41.790895 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:41.790904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.792015 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.792019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:41.793154 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:41.793169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.794236 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.794241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:41.794376 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:41.801154 (+ 6778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:41.801176 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:41.804188 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":419,"threads_started":1}
W20260620 14:54:41.804838 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:41.789136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:56579 (local address 127.21.159.254:38017)
0620 14:54:41.789307 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:41.789313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:41.789890 (+ 577us) server_negotiation.cc:407] Connection header received
0620 14:54:41.790036 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:41.790048 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:41.790117 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:41.790216 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:41.791045 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.791864 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.793320 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:41.794065 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:41.794164 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:41.804525 (+ 10361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:56579: BlockingRecv error: recv got EOF from 127.21.159.194:56579 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:54:42 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:42 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:42.281858 31436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:42.273034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:47289)
0620 14:54:42.273580 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:42.273600 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:42.273760 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:42.274126 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:42.274131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:42.274526 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:42.274679 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:42.274688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.275795 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.275802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:42.276712 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:42.276723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.277489 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.277495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:42.277667 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:42.278315 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:42.278333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:42.281616 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":301,"threads_started":1}
W20260620 14:54:42.282159 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:42.273128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:47289 (local address 127.21.159.254:38017)
0620 14:54:42.273363 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:42.273370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:42.273686 (+ 316us) server_negotiation.cc:407] Connection header received
0620 14:54:42.273885 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:42.273894 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:42.273977 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:42.274086 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:42.274825 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.275651 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.276860 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.277316 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.277457 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:42.281870 (+ 4413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:47289: BlockingRecv error: recv got EOF from 127.21.159.195:47289 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 14:54:42 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:42 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:42.310824 31437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:42.301602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:39149)
0620 14:54:42.302187 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:42.302203 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:42.302333 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:42.302645 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:42.302648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:42.303048 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:42.303287 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:42.303297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.304522 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.304526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:42.305461 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:42.305473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.306566 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.306572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:42.306729 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:42.307458 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:42.307477 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:42.310568 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":293,"threads_started":1}
W20260620 14:54:42.311139 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:42.301721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:39149 (local address 127.21.159.254:38017)
0620 14:54:42.301937 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:42.301942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:42.302299 (+ 357us) server_negotiation.cc:407] Connection header received
0620 14:54:42.302446 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:42.302450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:42.302515 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:42.302601 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:42.303481 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.304308 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.305608 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.306290 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.306475 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:42.310830 (+ 4355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:39149: BlockingRecv error: recv got EOF from 127.21.159.193:39149 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:54:42 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:42 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:42.815403 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:42.805634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:38067)
0620 14:54:42.806354 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:42.806371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:42.806580 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:42.806991 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:42.806996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:42.807487 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:42.807727 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:42.807740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.809206 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.809211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:42.810288 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:42.810301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.811082 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.811087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:42.811240 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:42.811896 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:42.811914 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:42.815023 (+ 3109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":286,"threads_started":1}
W20260620 14:54:42.815544 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:42.805756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:38067 (local address 127.21.159.254:38017)
0620 14:54:42.806034 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:42.806042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:42.806492 (+ 450us) server_negotiation.cc:407] Connection header received
0620 14:54:42.806757 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:42.806764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:42.806845 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:42.806958 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:42.807897 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.809033 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.810417 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:42.810894 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:42.810995 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:42.815309 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:38067: BlockingRecv error: recv got EOF from 127.21.159.194:38067 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Jun 20 14:54:43 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:43 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:43.292261 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:43.283121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:37655)
0620 14:54:43.283735 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:43.283756 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:43.283882 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:43.284197 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:43.284200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:43.284598 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:43.284790 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:43.284800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.285829 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.285832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:43.286800 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:43.286835 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.287884 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.287889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:43.288005 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:43.288601 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:43.288618 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:43.292006 (+ 3388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":286,"threads_started":1}
W20260620 14:54:43.292505 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:43.283203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:37655 (local address 127.21.159.254:38017)
0620 14:54:43.283432 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:43.283441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:43.283882 (+ 441us) server_negotiation.cc:407] Connection header received
0620 14:54:43.283972 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:43.283976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:43.284041 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:43.284141 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:43.284955 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.285703 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.286974 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.287725 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.287824 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:43.292282 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:37655: BlockingRecv error: recv got EOF from 127.21.159.195:37655 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 14:54:43 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:43 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:43.320863 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:43.311853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:50911)
0620 14:54:43.312425 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:43.312443 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:43.312587 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:43.312950 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:43.312957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:43.313388 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:43.313608 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:43.313621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.314994 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.315002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:43.316009 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:43.316021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.316863 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.316869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:43.317058 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:43.317735 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:43.317753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:43.320590 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":298,"threads_started":1}
W20260620 14:54:43.321149 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:43.312090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:50911 (local address 127.21.159.254:38017)
0620 14:54:43.312293 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:43.312298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:43.312530 (+ 232us) server_negotiation.cc:407] Connection header received
0620 14:54:43.312712 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:43.312716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:43.312779 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:43.312872 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:43.313770 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.314770 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.316159 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.316652 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.316767 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:43.320814 (+ 4047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:50911: BlockingRecv error: recv got EOF from 127.21.159.193:50911 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:54:43 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:43 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:43.826362 31441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:43.816775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:52627)
0620 14:54:43.817440 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:43.817459 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:43.817642 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:43.818010 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:43.818015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:43.818461 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:43.818615 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:43.818625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.820017 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.820025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:43.821057 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:43.821069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.821887 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.821895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:43.822075 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:43.822760 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:43.822780 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:43.826101 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":299,"threads_started":1}
W20260620 14:54:43.826766 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:43.816907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:52627 (local address 127.21.159.254:38017)
0620 14:54:43.817159 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:43.817166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:43.817572 (+ 406us) server_negotiation.cc:407] Connection header received
0620 14:54:43.817794 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:43.817804 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:43.817868 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:43.817981 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:43.818786 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.819857 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.821209 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:43.821685 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:43.821842 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:43.826326 (+ 4484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:52627: BlockingRecv error: recv got EOF from 127.21.159.194:52627 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 14:54:44 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:44 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:44.302304 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:44.293440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:50713)
0620 14:54:44.294022 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:44.294047 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:44.294202 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:44.294669 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:44.294674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:44.295242 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:44.295396 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:44.295405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.296428 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.296432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:44.297383 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:44.297394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.298105 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.298110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:44.298227 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:44.298909 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:44.298927 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:44.302055 (+ 3128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":496,"threads_started":1}
W20260620 14:54:44.302631 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:44.293708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:50713 (local address 127.21.159.254:38017)
0620 14:54:44.293930 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:44.293937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:44.294142 (+ 205us) server_negotiation.cc:407] Connection header received
0620 14:54:44.294348 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:44.294356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:44.294470 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:44.294630 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:44.295548 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.296272 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.297510 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.297924 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.298018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:44.302288 (+ 4270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:50713: BlockingRecv error: recv got EOF from 127.21.159.195:50713 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:54:44 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:44 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:44.331239 31443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:44.321824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:60705)
0620 14:54:44.322333 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:44.322350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:44.322466 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:44.322807 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:44.322811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:44.323365 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:44.323536 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:44.323545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.324784 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.324789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:44.325879 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:44.325892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.326792 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.326799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:44.326999 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:44.327619 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:44.327638 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:44.330821 (+ 3183us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":248,"threads_started":1}
W20260620 14:54:44.331363 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:44.321872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:60705 (local address 127.21.159.254:38017)
0620 14:54:44.322113 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:44.322120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:44.322452 (+ 332us) server_negotiation.cc:407] Connection header received
0620 14:54:44.322570 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:44.322574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:44.322634 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:44.322716 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:44.323668 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.324622 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.326052 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.326570 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.326776 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:44.331097 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:60705: BlockingRecv error: recv got EOF from 127.21.159.193:60705 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:54:44 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:44 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:44.837231 31444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:44.827390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:46739)
0620 14:54:44.827965 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:44.827983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:44.828109 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:44.828543 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:44.828547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:44.828964 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:44.829176 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:44.829189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.830696 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.830701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:44.831856 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:44.831872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.832655 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.832660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:44.832810 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:44.833462 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:44.833493 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:44.836811 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":325,"threads_started":1}
W20260620 14:54:44.837339 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:44.827514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:46739 (local address 127.21.159.254:38017)
0620 14:54:44.827705 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:44.827710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:44.828087 (+ 377us) server_negotiation.cc:407] Connection header received
0620 14:54:44.828220 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:44.828226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:44.828325 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:44.828448 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:44.829398 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.830508 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.832022 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:44.832454 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:44.832563 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:44.837064 (+ 4501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:46739: BlockingRecv error: recv got EOF from 127.21.159.194:46739 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:54:45 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:45 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:45.312443 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:45.303448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:59931)
0620 14:54:45.304010 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:45.304031 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:45.304239 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:45.304598 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:45.304603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:45.305037 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:45.305207 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:45.305216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.306299 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.306307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:45.307539 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:45.307552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.308282 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.308288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:45.308470 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:45.309136 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:45.309155 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:45.312079 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":310,"threads_started":1}
W20260620 14:54:45.312523 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:45.303573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:59931 (local address 127.21.159.254:38017)
0620 14:54:45.303810 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:45.303820 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:45.304127 (+ 307us) server_negotiation.cc:407] Connection header received
0620 14:54:45.304374 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:45.304380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:45.304454 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:45.304566 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:45.305364 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.306165 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.307683 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.308123 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.308269 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:45.312276 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:59931: BlockingRecv error: recv got EOF from 127.21.159.195:59931 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 14:54:45 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:45 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:45.341441 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:45.332296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:34691)
0620 14:54:45.332841 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:45.332857 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:45.333013 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:45.333388 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:45.333393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:45.333786 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:45.333995 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:45.334005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.335228 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.335234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:45.336191 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:45.336204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.336969 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.336975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:45.337239 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:45.338030 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:45.338050 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:45.341193 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":305,"threads_started":1}
W20260620 14:54:45.341681 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:45.332338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:34691 (local address 127.21.159.254:38017)
0620 14:54:45.332537 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:45.332541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:45.332946 (+ 405us) server_negotiation.cc:407] Connection header received
0620 14:54:45.333153 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:45.333158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:45.333244 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:45.333363 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:45.334189 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.334977 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.336324 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.336748 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.336846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:45.341426 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:34691: BlockingRecv error: recv got EOF from 127.21.159.193:34691 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 14:54:45 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:45 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:45.847452 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:45.838309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:37383)
0620 14:54:45.838876 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:45.838893 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:45.839013 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:45.839456 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:45.839460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:45.839860 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:45.840012 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:45.840020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.841002 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.841006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:45.841957 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:45.841968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.842933 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.842938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:45.843126 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:45.843782 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:45.843800 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:45.847217 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":323,"threads_started":1}
W20260620 14:54:45.847659 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:45.838398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:37383 (local address 127.21.159.254:38017)
0620 14:54:45.838571 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:45.838576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:45.838992 (+ 416us) server_negotiation.cc:407] Connection header received
0620 14:54:45.839179 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:45.839188 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:45.839278 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:45.839407 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:45.840163 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.840866 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.842124 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:45.842784 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:45.842889 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:45.847392 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:37383: BlockingRecv error: recv got EOF from 127.21.159.194:37383 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:54:46 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:46 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:46.322427 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:46.313739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:46979)
0620 14:54:46.314360 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:46.314383 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:46.314566 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:46.314915 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:46.314920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:46.315388 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:46.315549 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:46.315558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.316601 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.316608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:46.317530 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:46.317541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.318264 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.318270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:46.318445 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:46.319157 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:46.319174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:46.322181 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":390,"threads_started":1}
W20260620 14:54:46.322691 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:46.314058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:46979 (local address 127.21.159.254:38017)
0620 14:54:46.314354 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:46.314361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:46.314497 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:54:46.314703 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:46.314708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:46.314775 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:46.314881 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:46.315721 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.316465 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.317677 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.318105 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.318248 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:46.322390 (+ 4142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:46979: BlockingRecv error: recv got EOF from 127.21.159.195:46979 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Jun 20 14:54:46 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:46 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:46.351917 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:46.342561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:46775)
0620 14:54:46.343135 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:46.343154 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:46.343276 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:46.343598 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:46.343601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:46.344019 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:46.344196 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:46.344205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.345213 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.345217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:46.346094 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:46.346104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.346907 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.346912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:46.347037 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:46.347834 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:46.347861 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:46.351544 (+ 3683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
W20260620 14:54:46.352367 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:46.342678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:46775 (local address 127.21.159.254:38017)
0620 14:54:46.342982 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:46.342992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:46.343263 (+ 271us) server_negotiation.cc:407] Connection header received
0620 14:54:46.343384 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:46.343389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:46.343472 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:46.343571 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:46.344348 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.345093 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.346258 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.346755 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.346869 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:46.351813 (+ 4944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:46775: BlockingRecv error: recv got EOF from 127.21.159.193:46775 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Jun 20 14:54:46 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:46 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:46.861346 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:46.848387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:34751)
0620 14:54:46.848976 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:46.848994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:46.849132 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:46.849453 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:46.849456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:46.849846 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:46.850013 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:46.850022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.851224 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.851228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:46.852367 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:46.852387 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.853148 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.853152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:46.853279 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:46.857927 (+ 4648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:46.857955 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:46.861095 (+ 3140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":356,"threads_started":1}
W20260620 14:54:46.861640 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:46.848574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:34751 (local address 127.21.159.254:38017)
0620 14:54:46.848746 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:46.848753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:46.849071 (+ 318us) server_negotiation.cc:407] Connection header received
0620 14:54:46.849251 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:46.849255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:46.849318 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:46.849412 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:46.850175 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.851086 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.852530 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:46.852994 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:46.853095 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:46.861336 (+ 8241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:34751: BlockingRecv error: recv got EOF from 127.21.159.194:34751 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:54:47 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:47 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:47.332067 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:47.323640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:57271)
0620 14:54:47.324264 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:47.324295 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:47.324440 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:47.324739 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:47.324743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:47.325176 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:47.325354 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:47.325363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.326362 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.326365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:47.327292 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:47.327304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.327972 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.327976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:47.328095 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:47.328699 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:47.328717 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:47.331802 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":347,"threads_started":1}
W20260620 14:54:47.332261 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:47.323802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:57271 (local address 127.21.159.254:38017)
0620 14:54:47.324067 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:47.324074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:47.324399 (+ 325us) server_negotiation.cc:407] Connection header received
0620 14:54:47.324539 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:47.324543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:47.324607 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:47.324693 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:47.325495 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.326243 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.327420 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.327817 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.327909 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:47.332009 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:57271: BlockingRecv error: recv got EOF from 127.21.159.195:57271 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 14:54:47 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:47 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:47.361995 31452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:47.352995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:41981)
0620 14:54:47.353562 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:47.353579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:47.353761 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:47.354178 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:47.354183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:47.354749 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:47.354948 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:47.354958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.356177 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.356181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:47.357127 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:47.357139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.357870 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.357875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:47.358003 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:47.358701 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:47.358718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:47.361643 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":312,"threads_started":1}
W20260620 14:54:47.362336 31154 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:47.353051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:41981 (local address 127.21.159.254:38017)
0620 14:54:47.353309 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:47.353316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:47.353709 (+ 393us) server_negotiation.cc:407] Connection header received
0620 14:54:47.353912 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:47.353918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:47.354017 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:47.354134 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:47.355135 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.356035 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.357272 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.357740 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.357841 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:47.361913 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:41981: BlockingRecv error: recv got EOF from 127.21.159.193:41981 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 14:54:47 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:47 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:47.871948 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:47.862585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:43705)
0620 14:54:47.863364 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:47.863389 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:47.863563 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:47.864278 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:47.864282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:47.864695 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:47.864859 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:47.864867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.865973 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.865977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:47.866930 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:47.866941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.867755 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.867761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:47.867980 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:47.868682 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:47.868699 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:47.871591 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":415,"threads_started":1}
W20260620 14:54:47.872128 31454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:47.862692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:43705 (local address 127.21.159.254:38017)
0620 14:54:47.863807 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:47.863818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:47.863851 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:54:47.863945 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:47.863953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:47.864044 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:47.864153 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:47.865043 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.865812 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.867128 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:47.867593 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:47.867741 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:47.871832 (+ 4091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:43705: BlockingRecv error: recv got EOF from 127.21.159.194:43705 (error 108)
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":714,"threads_started":1}
Jun 20 14:54:48 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:48 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:48.342514 31455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:48.333052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:38387)
0620 14:54:48.333638 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:48.333662 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:48.333784 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:48.334209 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:48.334214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:48.334598 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:48.334751 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:48.334760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.336142 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.336146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:48.337156 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:48.337169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.338037 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.338041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:48.338171 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:48.338839 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:48.338857 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:48.342131 (+ 3274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":388,"threads_started":1}
W20260620 14:54:48.342660 31454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:48.333099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:38387 (local address 127.21.159.254:38017)
0620 14:54:48.333312 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:48.333347 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:48.333840 (+ 493us) server_negotiation.cc:407] Connection header received
0620 14:54:48.333903 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:48.333908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:48.333975 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:48.334116 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:48.334950 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.335972 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.337301 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.337889 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.337993 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:48.342394 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:38387: BlockingRecv error: recv got EOF from 127.21.159.195:38387 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 14:54:48 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:48 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:48.372335 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:48.363179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:49359)
0620 14:54:48.363775 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:48.363800 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:48.363928 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:48.364293 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:48.364296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:48.364697 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:48.364907 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:48.364917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.366095 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.366099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:48.367116 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:48.367130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.368008 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.368016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:48.368183 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:48.368894 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:48.368912 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:48.372029 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
W20260620 14:54:48.372579 31454 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:48.363255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:49359 (local address 127.21.159.254:38017)
0620 14:54:48.363520 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:48.363525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:48.363920 (+ 395us) server_negotiation.cc:407] Connection header received
0620 14:54:48.364019 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:48.364023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:48.364098 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:48.364228 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:48.365083 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.365936 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.367316 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.367787 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.367933 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:48.372308 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:49359: BlockingRecv error: recv got EOF from 127.21.159.193:49359 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 20 14:54:48 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:48 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:48.882426 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:48.873179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:38219)
0620 14:54:48.873667 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:48.873685 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:48.873807 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:48.874313 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:48.874316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:48.874708 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:48.874871 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:48.874917 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.876256 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.876274 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:48.877428 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:48.877442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.878171 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.878175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:48.878313 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:48.878924 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:48.878943 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:48.882066 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
W20260620 14:54:48.882611 31458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:48.873500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:38219 (local address 127.21.159.254:38017)
0620 14:54:48.873974 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:48.873978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:48.873996 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:54:48.874053 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:48.874057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:48.874123 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:48.874223 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:48.875141 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.876106 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.877570 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:48.878005 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:48.878113 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:48.882311 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:38219: BlockingRecv error: recv got EOF from 127.21.159.194:38219 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":276,"threads_started":1}
Jun 20 14:54:49 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:49 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:49.353150 31459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:49.343834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:39071)
0620 14:54:49.344408 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:49.344428 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:49.344554 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:49.344898 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:49.344902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:49.345332 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:49.345491 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:49.345501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.346607 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.346611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:49.347665 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:49.347678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.348669 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.348676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:49.348864 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:49.349657 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:49.349682 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:49.352906 (+ 3224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":294,"threads_started":1}
W20260620 14:54:49.353471 31458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:49.343996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:39071 (local address 127.21.159.254:38017)
0620 14:54:49.344181 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:49.344186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:49.344563 (+ 377us) server_negotiation.cc:407] Connection header received
0620 14:54:49.344641 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:49.344652 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:49.344719 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:49.344814 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:49.345681 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.346440 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.347874 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.348446 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.348595 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:49.353160 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:39071: BlockingRecv error: recv got EOF from 127.21.159.195:39071 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:54:49 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:49 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:49.381999 31460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:49.373720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:58007)
0620 14:54:49.374273 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:49.374294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:49.374474 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:49.374889 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:49.374894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:49.375412 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:49.375587 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:49.375597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.376581 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.376585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:49.377475 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:49.377486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.378145 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.378149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:49.378264 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:49.378945 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:49.378962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:49.381696 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":284,"threads_started":1}
W20260620 14:54:49.382210 31458 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:49.373907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:58007 (local address 127.21.159.254:38017)
0620 14:54:49.374256 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:49.374262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:49.374394 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:54:49.374602 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:49.374607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:49.374718 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:49.374816 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:49.375722 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.376460 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.377599 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.378026 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.378118 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:49.381950 (+ 3832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:58007: BlockingRecv error: recv got EOF from 127.21.159.193:58007 (error 108)
Metrics: {"server-negotiator.queue_time_us":222}
Jun 20 14:54:49 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:49 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:49.893381 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:49.883601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:52235)
0620 14:54:49.884121 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:49.884148 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:49.884301 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:49.884784 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:49.884800 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:49.885198 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:49.885375 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:49.885384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.886469 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.886480 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:49.887486 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:49.887499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.888445 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.888454 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:49.888710 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:49.889544 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:49.889561 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:49.892981 (+ 3420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":304,"threads_started":1}
W20260620 14:54:49.893476 31462 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:49.883745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:52235 (local address 127.21.159.254:38017)
0620 14:54:49.884446 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:49.884451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:49.884466 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:54:49.884539 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:49.884542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:49.884620 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:49.884744 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:49.885538 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.886315 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.887642 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:49.888261 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:49.888408 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:49.893199 (+ 4791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:52235: BlockingRecv error: recv got EOF from 127.21.159.194:52235 (error 108)
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":426,"threads_started":1}
Jun 20 14:54:50 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:50 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.195@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:50.362267 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:50.354108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.195:59871)
0620 14:54:50.354575 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:50.354599 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:50.354767 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:50.355134 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:50.355140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:50.355518 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:50.355689 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:50.355697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.356712 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.356718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:50.357631 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:50.357641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.358342 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.358348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:50.358525 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:50.359155 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:50.359172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:50.361896 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
W20260620 14:54:50.362329 31462 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:50.354179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:59871 (local address 127.21.159.254:38017)
0620 14:54:50.354422 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:50.354430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:50.354695 (+ 265us) server_negotiation.cc:407] Connection header received
0620 14:54:50.354890 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:50.354895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:50.354962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:50.355085 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:50.355836 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.356566 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.357771 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.358178 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.358330 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:50.362102 (+ 3772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.195:59871: BlockingRecv error: recv got EOF from 127.21.159.195:59871 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 14:54:50 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:50 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.193@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:50.392762 31464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:50.383376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.193:40107)
0620 14:54:50.383999 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:50.384016 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:50.384193 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:50.384584 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:50.384589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:50.385028 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:50.385190 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:50.385199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.386238 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.386245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:50.387232 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:50.387244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.388012 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.388019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:50.388191 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:50.388923 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:50.388940 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:50.392322 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":325,"threads_started":1}
W20260620 14:54:50.392910 31462 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:50.383514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:40107 (local address 127.21.159.254:38017)
0620 14:54:50.383865 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:50.383872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:50.384126 (+ 254us) server_negotiation.cc:407] Connection header received
0620 14:54:50.384342 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:50.384348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:50.384438 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:50.384552 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:50.385356 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.386085 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.387404 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.387852 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.387993 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:50.392589 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.193:40107: BlockingRecv error: recv got EOF from 127.21.159.193:40107 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Jun 20 14:54:50 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:54:50 dist-test-slave-3mqx krb5kdc[30414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.159.194@KRBTEST.COM for oryx/127.21.159.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:54:50.902829 31465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:50.894448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38017 (local address 127.21.159.194:49471)
0620 14:54:50.894983 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:50.895001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:50.895173 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:50.895505 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:50.895508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:50.895891 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:50.896051 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:50.896059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.897081 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.897084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:50.898035 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:50.898047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.898805 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.898810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:50.898932 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:50.899707 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:50.899726 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:50.902488 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.159.254:38017: Server oryx/127.21.159.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
W20260620 14:54:50.903081 31466 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:50.894535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:49471 (local address 127.21.159.254:38017)
0620 14:54:50.895100 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:50.895107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:50.895130 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:54:50.895282 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:50.895285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:50.895354 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:50.895445 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:50.896185 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.896943 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.898166 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:50.898621 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:50.898730 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:50.902741 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.159.194:49471: BlockingRecv error: recv got EOF from 127.21.159.194:49471 (error 108)
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
I20260620 14:54:51.188827 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31003
I20260620 14:54:51.206462 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:51.189941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.21.159.193:42923)
0620 14:54:51.190549 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:51.190557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:51.190585 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:54:51.190655 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:51.190661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:51.190892 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:51.191128 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:51.192593 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.193714 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.194960 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.195992 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.196117 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:51.203155 (+ 7038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:51.203165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:51.203181 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:51.203219 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:51.205748 (+ 2529us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.159.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 14:54:51.206129 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60618: Request ticket server kudu/127.21.159.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":312,"threads_started":1}
W20260620 14:54:51.206580 31468 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60618: Request ticket server kudu/127.21.159.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 14:54:51.206573 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31003: Not authorized: Client connection negotiation failed: client connection to 127.21.159.193:42923: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.159.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 14:54:51.206842 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31003
I20260620 14:54:51.224361 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31155
I20260620 14:54:51.237005 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:51.225503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.21.159.194:34449)
0620 14:54:51.226177 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:51.226184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:51.226217 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:54:51.226277 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:51.226283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:51.226518 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:51.226668 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:51.228031 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.229087 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.230355 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.230935 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.231037 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:51.233613 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:51.233621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:51.233636 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:51.233676 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:51.236389 (+ 2713us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.159.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 14:54:51.236742 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60118: Request ticket server kudu/127.21.159.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":359,"threads_started":1}
W20260620 14:54:51.237135 31469 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60118: Request ticket server kudu/127.21.159.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 14:54:51.237222 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31155: Not authorized: Client connection negotiation failed: client connection to 127.21.159.194:34449: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.159.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 14:54:51.237426 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31155
I20260620 14:54:51.254154 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31286
I20260620 14:54:51.266579 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:51.255212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.21.159.195:44309)
0620 14:54:51.255818 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:51.255826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:51.255853 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:54:51.255913 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:51.255920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:51.256144 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:51.256318 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:51.257563 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.258611 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.259967 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.260573 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.260675 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:51.263320 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:51.263329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:51.263344 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:51.263387 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:51.266021 (+ 2634us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.159.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 14:54:51.266346 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35782: Request ticket server kudu/127.21.159.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":330,"threads_started":1}
W20260620 14:54:51.266716 31470 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35782: Request ticket server kudu/127.21.159.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 14:54:51.266788 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31286: Not authorized: Client connection negotiation failed: client connection to 127.21.159.195:44309: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.21.159.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 14:54:51.266984 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31286
I20260620 14:54:51.283342 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 30934
I20260620 14:54:51.284641 30996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:51.410555 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 30934
2026-06-20T14:54:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21520 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:51Z Disabled control of system clock
I20260620 14:54:51.505007 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42835
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:42835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:54:51.848026 31475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:51.848336 31475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:51.848518 31475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:51.856330 31475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:51.856469 31475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:51.856567 31475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:51.856671 31475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:51.856765 31475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:51.866400 31475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:42835
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:51.867904 31475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:51.869884 31475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:51.883066 31480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:51.882958 31483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:51.883044 31481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:51.883883 31475 server_base.cc:1034] running on GCE node
I20260620 14:54:51.884850 31475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:51.886524 31475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:51.887794 31475 hybrid_clock.cc:648] HybridClock initialized: now 1781967291887730 us; error 67 us; skew 500 ppm
I20260620 14:54:51.888213 31475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:51.894492 31475 webserver.cc:460] Webserver started at http://127.21.159.254:36223/ using document root <none> and password file <none>
I20260620 14:54:51.895483 31475 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:51.895648 31475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:51.896045 31475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:51.899681 31475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "58448cfea6084768887ec363c1f925fd"
format_stamp: "Formatted at 2026-06-20 14:54:51 on dist-test-slave-3mqx"
server_key: "d5bcf5116a6e69e6d1d9cf3b0d13b039"
server_key_iv: "2e83419208c485c8866503398256053a"
server_key_version: "encryptionkey@0"
I20260620 14:54:51.900650 31475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "58448cfea6084768887ec363c1f925fd"
format_stamp: "Formatted at 2026-06-20 14:54:51 on dist-test-slave-3mqx"
server_key: "d5bcf5116a6e69e6d1d9cf3b0d13b039"
server_key_iv: "2e83419208c485c8866503398256053a"
server_key_version: "encryptionkey@0"
I20260620 14:54:51.907038 31475 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:54:51.911206 31489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:51.912961 31475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:54:51.913187 31475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "58448cfea6084768887ec363c1f925fd"
format_stamp: "Formatted at 2026-06-20 14:54:51 on dist-test-slave-3mqx"
server_key: "d5bcf5116a6e69e6d1d9cf3b0d13b039"
server_key_iv: "2e83419208c485c8866503398256053a"
server_key_version: "encryptionkey@0"
I20260620 14:54:51.913513 31475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:51.938061 31475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:51.939116 31475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:51.939451 31475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:51.959323 31475 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:42835
I20260620 14:54:51.959342 31540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:42835 every 8 connection(s)
I20260620 14:54:51.960796 31475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:51.965291 31541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:51.969414 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31475
I20260620 14:54:51.969655 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:51.970319 22143 external_mini_cluster.cc:1428] Setting key ff96df3b404443ccfbf3e51127399a13
I20260620 14:54:51.979653 31541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd: Bootstrap starting.
I20260620 14:54:51.983438 31541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:51.984748 31541 log.cc:824] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:51.984953 31544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:51.972648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.21.159.254:42835)
0620 14:54:51.973818 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:51.973830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:51.973874 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:54:51.975660 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:51.975688 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:51.976260 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:51.976737 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:51.977418 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.978749 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.980220 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:51.980936 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:51.981151 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:51.982473 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:51.982484 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:51.982503 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:51.983131 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:51.983229 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:51.983251 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:51.983386 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:51.983614 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:54:51.983944 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":415,"threads_started":1}
I20260620 14:54:51.988462 31541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd: No bootstrap required, opened a new log
I20260620 14:54:51.996858 31541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58448cfea6084768887ec363c1f925fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42835 } }
I20260620 14:54:51.997342 31541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:51.997486 31541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58448cfea6084768887ec363c1f925fd, State: Initialized, Role: FOLLOWER
I20260620 14:54:51.998234 31541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58448cfea6084768887ec363c1f925fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42835 } }
I20260620 14:54:51.998508 31541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:51.998674 31541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:51.998886 31541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:52.001267 31541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58448cfea6084768887ec363c1f925fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42835 } }
I20260620 14:54:52.001852 31541 leader_election.cc:302] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58448cfea6084768887ec363c1f925fd; no voters:
I20260620 14:54:52.002525 31541 leader_election.cc:288] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:52.002763 31546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:52.003947 31546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [term 1 LEADER]: Becoming Leader. State: Replica: 58448cfea6084768887ec363c1f925fd, State: Running, Role: LEADER
I20260620 14:54:52.004503 31546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58448cfea6084768887ec363c1f925fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42835 } }
I20260620 14:54:52.005054 31541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:52.007548 31548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58448cfea6084768887ec363c1f925fd. Latest consensus state: current_term: 1 leader_uuid: "58448cfea6084768887ec363c1f925fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58448cfea6084768887ec363c1f925fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42835 } } }
I20260620 14:54:52.007790 31547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58448cfea6084768887ec363c1f925fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58448cfea6084768887ec363c1f925fd" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42835 } } }
I20260620 14:54:52.008081 31548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:52.008124 31547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:52.010059 31555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:52.016345 31555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:52.027119 31555 catalog_manager.cc:1347] Generated new cluster ID: 2882e9788d0b4f4a98eb6999b45edfcf
I20260620 14:54:52.027302 31555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:52.037103 31555 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:52.039095 31555 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:52.052779 31555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58448cfea6084768887ec363c1f925fd: Generated new TSK 0
I20260620 14:54:52.054085 31555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:52.060907 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42835
--builtin_ntp_servers=127.21.159.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:54:52.419623 31565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:52.419987 31565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:52.420146 31565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:52.427835 31565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:52.427995 31565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:52.428184 31565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:52.437634 31565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:52.439036 31565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:52.440973 31565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:52.454821 31571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:52.454804 31573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:52.454846 31570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:52.456174 31565 server_base.cc:1034] running on GCE node
I20260620 14:54:52.457062 31565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:52.458159 31565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:52.459409 31565 hybrid_clock.cc:648] HybridClock initialized: now 1781967292459358 us; error 52 us; skew 500 ppm
I20260620 14:54:52.459775 31565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:52.462538 31565 webserver.cc:460] Webserver started at http://127.21.159.193:39797/ using document root <none> and password file <none>
I20260620 14:54:52.463411 31565 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:52.463551 31565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:52.463910 31565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:52.467042 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "b84ac78b46cb4013b4886ec3e6070671"
format_stamp: "Formatted at 2026-06-20 14:54:52 on dist-test-slave-3mqx"
server_key: "e345af0c0bc4a2c013596409e7463097"
server_key_iv: "c192288f1937eec2a143c509976c30c0"
server_key_version: "encryptionkey@0"
I20260620 14:54:52.468009 31565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "b84ac78b46cb4013b4886ec3e6070671"
format_stamp: "Formatted at 2026-06-20 14:54:52 on dist-test-slave-3mqx"
server_key: "e345af0c0bc4a2c013596409e7463097"
server_key_iv: "c192288f1937eec2a143c509976c30c0"
server_key_version: "encryptionkey@0"
I20260620 14:54:52.474066 31565 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260620 14:54:52.478227 31579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:52.479941 31565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:52.480104 31565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "b84ac78b46cb4013b4886ec3e6070671"
format_stamp: "Formatted at 2026-06-20 14:54:52 on dist-test-slave-3mqx"
server_key: "e345af0c0bc4a2c013596409e7463097"
server_key_iv: "c192288f1937eec2a143c509976c30c0"
server_key_version: "encryptionkey@0"
I20260620 14:54:52.480312 31565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:52.504115 31565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:52.505395 31565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:52.505789 31565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:52.507014 31565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:52.508747 31565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:52.508867 31565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:52.508984 31565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:52.509052 31565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:52.545843 31565 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:33255
I20260620 14:54:52.545948 31691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:33255 every 8 connection(s)
I20260620 14:54:52.547284 31565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:52.548017 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31565
I20260620 14:54:52.548280 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:52.549202 22143 external_mini_cluster.cc:1428] Setting key c96f852621ee88ea39734e23cd6c1abd
I20260620 14:54:52.552663 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42835
--builtin_ntp_servers=127.21.159.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:54:52.562668 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:52.550315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:58627 (local address 127.21.159.254:42835)
0620 14:54:52.550803 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:52.550809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:52.553047 (+ 2238us) server_negotiation.cc:407] Connection header received
0620 14:54:52.554607 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:52.554616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:52.554715 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:52.554855 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:52.556187 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:52.557486 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:52.559316 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:52.559809 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:52.559925 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:52.561805 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:52.561810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:52.561814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:52.561835 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:52.561875 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:52.561887 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:52.561965 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:52.562249 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:54:52.562332 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260620 14:54:52.564461 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:52.551045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42835 (local address 127.21.159.193:58627)
0620 14:54:52.552329 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:52.552378 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:52.554297 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:52.555212 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:52.555221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:52.555338 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:52.555998 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:52.556017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:52.557676 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:52.557680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:52.559146 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:52.559165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:52.560085 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:52.560094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:52.560461 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:52.561335 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:52.561341 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:52.561400 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:52.561528 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:52.561995 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:52.562009 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:52.562020 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:54:52.562208 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:54:52.562852 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":412,"threads_started":1}
I20260620 14:54:52.566320 31692 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42835
I20260620 14:54:52.566702 31692 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:52.567658 31692 heartbeater.cc:511] Master 127.21.159.254:42835 requested a full tablet report, sending...
I20260620 14:54:52.570013 31506 ts_manager.cc:194] Registered new tserver with Master: b84ac78b46cb4013b4886ec3e6070671 (127.21.159.193:33255)
I20260620 14:54:52.571744 31506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.193:58627
W20260620 14:54:52.911813 31696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:52.912140 31696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:52.912297 31696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:52.920881 31696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:52.921002 31696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:52.921172 31696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:52.930694 31696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:52.932101 31696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:52.933946 31696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:52.948266 31701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:52.948309 31702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:52.948344 31704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:52.949349 31696 server_base.cc:1034] running on GCE node
I20260620 14:54:52.950160 31696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:52.951303 31696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:52.952559 31696 hybrid_clock.cc:648] HybridClock initialized: now 1781967292952503 us; error 53 us; skew 500 ppm
I20260620 14:54:52.952924 31696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:52.955636 31696 webserver.cc:460] Webserver started at http://127.21.159.194:41137/ using document root <none> and password file <none>
I20260620 14:54:52.956405 31696 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:52.956547 31696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:52.956877 31696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:52.960127 31696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "dbb2591149f142c4a9d7770265fff79d"
format_stamp: "Formatted at 2026-06-20 14:54:52 on dist-test-slave-3mqx"
server_key: "41780968e92c08573d19742d3bb40805"
server_key_iv: "9baba410856e3bf309183f1e2b684778"
server_key_version: "encryptionkey@0"
I20260620 14:54:52.960973 31696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "dbb2591149f142c4a9d7770265fff79d"
format_stamp: "Formatted at 2026-06-20 14:54:52 on dist-test-slave-3mqx"
server_key: "41780968e92c08573d19742d3bb40805"
server_key_iv: "9baba410856e3bf309183f1e2b684778"
server_key_version: "encryptionkey@0"
I20260620 14:54:52.966434 31696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 14:54:52.970351 31710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:52.972026 31696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:52.972227 31696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "dbb2591149f142c4a9d7770265fff79d"
format_stamp: "Formatted at 2026-06-20 14:54:52 on dist-test-slave-3mqx"
server_key: "41780968e92c08573d19742d3bb40805"
server_key_iv: "9baba410856e3bf309183f1e2b684778"
server_key_version: "encryptionkey@0"
I20260620 14:54:52.972479 31696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:52.992404 31696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:52.993341 31696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:52.993678 31696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:52.994774 31696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:52.996392 31696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:52.996523 31696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:52.996665 31696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:52.996752 31696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:53.032542 31696 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:42753
I20260620 14:54:53.032584 31822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:42753 every 8 connection(s)
I20260620 14:54:53.033977 31696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:53.040783 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31696
I20260620 14:54:53.041005 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:53.041554 22143 external_mini_cluster.cc:1428] Setting key 6b522342c306227d17335e07119e222f
I20260620 14:54:53.044641 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42835
--builtin_ntp_servers=127.21.159.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:54:53.048010 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.036825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:46923 (local address 127.21.159.254:42835)
0620 14:54:53.037033 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:53.037039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:53.038630 (+ 1591us) server_negotiation.cc:407] Connection header received
0620 14:54:53.040569 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:53.040574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:53.040648 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:53.040741 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:53.042427 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.043264 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.045134 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.045535 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.045673 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:53.047195 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:53.047201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:53.047204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:53.047223 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:53.047254 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:53.047266 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:53.047368 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:53.047596 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:54:53.047705 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:54:53.049162 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.037430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42835 (local address 127.21.159.194:46923)
0620 14:54:53.038446 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:53.038485 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:53.040185 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:53.041075 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:53.041089 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:53.041224 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:53.042165 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:53.042187 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.043418 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.043424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:53.044982 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:53.044999 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.045806 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.045822 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:53.046044 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:53.046809 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:53.046812 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:53.046857 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:53.046954 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:53.047367 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:53.047379 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:53.047389 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:54:53.047591 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 14:54:53.048046 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":317,"threads_started":1}
I20260620 14:54:53.050935 31823 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42835
I20260620 14:54:53.051280 31823 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:53.052095 31823 heartbeater.cc:511] Master 127.21.159.254:42835 requested a full tablet report, sending...
I20260620 14:54:53.053740 31506 ts_manager.cc:194] Registered new tserver with Master: dbb2591149f142c4a9d7770265fff79d (127.21.159.194:42753)
I20260620 14:54:53.054987 31506 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.194:46923
W20260620 14:54:53.389175 31826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:53.389508 31826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:53.389595 31826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:53.397081 31826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:53.397207 31826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:53.397315 31826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:53.406873 31826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:53.408324 31826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:53.410158 31826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:53.424324 31834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:53.424273 31832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:53.424470 31831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:53.425595 31826 server_base.cc:1034] running on GCE node
I20260620 14:54:53.426208 31826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:53.427358 31826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:53.428611 31826 hybrid_clock.cc:648] HybridClock initialized: now 1781967293428550 us; error 56 us; skew 500 ppm
I20260620 14:54:53.428982 31826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:53.431416 31826 webserver.cc:460] Webserver started at http://127.21.159.195:39705/ using document root <none> and password file <none>
I20260620 14:54:53.432192 31826 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:53.432343 31826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:53.432680 31826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:53.435755 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "2ab45bdad8c546dfab84e569c3aca86a"
format_stamp: "Formatted at 2026-06-20 14:54:53 on dist-test-slave-3mqx"
server_key: "891bf2a55c22e83c775f9aa30bcf16c8"
server_key_iv: "70277b31250dbfd2406ae9ffe7202d64"
server_key_version: "encryptionkey@0"
I20260620 14:54:53.436645 31826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "2ab45bdad8c546dfab84e569c3aca86a"
format_stamp: "Formatted at 2026-06-20 14:54:53 on dist-test-slave-3mqx"
server_key: "891bf2a55c22e83c775f9aa30bcf16c8"
server_key_iv: "70277b31250dbfd2406ae9ffe7202d64"
server_key_version: "encryptionkey@0"
I20260620 14:54:53.442538 31826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:54:53.447000 31840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:53.448583 31826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:54:53.448798 31826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "2ab45bdad8c546dfab84e569c3aca86a"
format_stamp: "Formatted at 2026-06-20 14:54:53 on dist-test-slave-3mqx"
server_key: "891bf2a55c22e83c775f9aa30bcf16c8"
server_key_iv: "70277b31250dbfd2406ae9ffe7202d64"
server_key_version: "encryptionkey@0"
I20260620 14:54:53.449038 31826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:53.474298 31826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:53.475381 31826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:53.475714 31826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:53.476877 31826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:53.478560 31826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:53.478698 31826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:53.478832 31826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:53.478924 31826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:53.514658 31826 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:34363
I20260620 14:54:53.514760 31952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:34363 every 8 connection(s)
I20260620 14:54:53.516206 31826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:53.520701 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31826
I20260620 14:54:53.520874 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:53.521373 22143 external_mini_cluster.cc:1428] Setting key a331d88f7608c2165d75b08921e53ce2
I20260620 14:54:53.529973 31694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.519149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:33153 (local address 127.21.159.254:42835)
0620 14:54:53.519373 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:53.519378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:53.521117 (+ 1739us) server_negotiation.cc:407] Connection header received
0620 14:54:53.523135 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:53.523141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:53.523228 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:53.523364 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:53.524524 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.525260 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.526855 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.527346 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.527486 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:53.529006 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:53.529012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:53.529015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:53.529037 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:53.529071 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:53.529084 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:53.529198 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:53.529470 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:54:53.529582 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:54:53.531344 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.519832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42835 (local address 127.21.159.195:33153)
0620 14:54:53.520928 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:53.520980 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:53.522837 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:53.523618 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:53.523627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:53.523736 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:53.524361 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:53.524378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.525429 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.525433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:53.526697 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:53.526708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.527580 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.527587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:53.527835 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:53.528609 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:53.528612 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:53.528659 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:53.528756 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:53.529209 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:53.529223 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:53.529235 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:54:53.529458 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 14:54:53.529929 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":300,"threads_started":1}
I20260620 14:54:53.532876 31953 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42835
I20260620 14:54:53.533219 31953 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:53.534032 31953 heartbeater.cc:511] Master 127.21.159.254:42835 requested a full tablet report, sending...
I20260620 14:54:53.535935 31505 ts_manager.cc:194] Registered new tserver with Master: 2ab45bdad8c546dfab84e569c3aca86a (127.21.159.195:34363)
I20260620 14:54:53.537004 31505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.195:33153
I20260620 14:54:53.537108 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:54:53.544006 31694 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:53.542122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.21.159.254:42835)
0620 14:54:53.542411 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:53.542421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:53.542866 (+ 445us) server_negotiation.cc:407] Connection header received
0620 14:54:53.543047 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:53.543114 (+ 67us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:53.543197 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:53.543290 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:53.543781 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49598: BlockingRecv error: recv got EOF from 127.0.0.1:49598 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 14:54:53.544418 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31565
I20260620 14:54:53.554011 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.545582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.21.159.193:33255)
0620 14:54:53.546413 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:53.546419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:53.546447 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:54:53.546504 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:53.546513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:53.546726 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:53.546864 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:53.547403 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.548485 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.550414 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.551031 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.551219 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:53.552452 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:53.552460 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:53.552473 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:53.553039 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:53.553088 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:53.553130 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:53.553228 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:53.553439 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:54:53.553609 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":555,"threads_started":1}
I20260620 14:54:53.555511 31687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:53.638912 31692 heartbeater.cc:503] Master 127.21.159.254:42835 was elected leader, sending a full tablet report...
I20260620 14:54:53.640094 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31565
W20260620 14:54:53.658600 31490 connection.cc:535] server connection from 127.21.159.193:58627 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:54:53.658988 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31696
I20260620 14:54:53.667851 31964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.660081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.21.159.194:42753)
0620 14:54:53.660731 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:53.660739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:53.660769 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:54:53.660838 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:53.660844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:53.661101 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:53.661238 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:53.661652 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.662666 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.664166 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.664769 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.664897 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:53.666116 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:53.666125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:53.666140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:53.666807 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:53.666872 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:53.666894 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:53.666992 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:53.667269 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:54:53.667407 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":371,"threads_started":1}
I20260620 14:54:53.669229 31818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:53.753253 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31696
I20260620 14:54:53.769654 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31826
I20260620 14:54:53.779417 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.770752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.21.159.195:34363)
0620 14:54:53.771599 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:53.771614 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:53.771650 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:54:53.771722 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:53.771729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:53.771969 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:53.772115 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:53.772691 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.773904 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.775541 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:53.776151 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:53.776274 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:53.777552 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:53.777562 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:53.777585 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:53.778250 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:53.778302 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:53.778330 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:53.778425 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:53.778687 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 14:54:53.778955 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":486,"threads_started":1}
I20260620 14:54:53.780900 31948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:53.864398 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31826
I20260620 14:54:53.880366 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31475
I20260620 14:54:53.881632 31536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:53.984020 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31475
2026-06-20T14:54:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2554 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:54Z Disabled control of system clock
I20260620 14:54:54.059253 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:34685
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:54:54.404891 31972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:54.405241 31972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:54.405417 31972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:54.413125 31972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:54.413247 31972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:54.413339 31972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:54.413441 31972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:54.413499 31972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:54.423878 31972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:34685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:54.425305 31972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:54.427265 31972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:54.439859 31977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:54.439857 31980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:54.439881 31978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:54.440778 31972 server_base.cc:1034] running on GCE node
I20260620 14:54:54.441664 31972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:54.443311 31972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:54.444600 31972 hybrid_clock.cc:648] HybridClock initialized: now 1781967294444502 us; error 60 us; skew 500 ppm
I20260620 14:54:54.444983 31972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:54.447517 31972 webserver.cc:460] Webserver started at http://127.21.159.254:38183/ using document root <none> and password file <none>
I20260620 14:54:54.448361 31972 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:54.448500 31972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:54.448861 31972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:54.451959 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "072e266f052948329d14bfd287a3bc8e"
format_stamp: "Formatted at 2026-06-20 14:54:54 on dist-test-slave-3mqx"
server_key: "3896a5788302c099e73b52f00338af79"
server_key_iv: "d63084fffbb97b71925990fd236f1f57"
server_key_version: "encryptionkey@0"
I20260620 14:54:54.452811 31972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "072e266f052948329d14bfd287a3bc8e"
format_stamp: "Formatted at 2026-06-20 14:54:54 on dist-test-slave-3mqx"
server_key: "3896a5788302c099e73b52f00338af79"
server_key_iv: "d63084fffbb97b71925990fd236f1f57"
server_key_version: "encryptionkey@0"
I20260620 14:54:54.458245 31972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 14:54:54.462342 31986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:54.463869 31972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:54.464071 31972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "072e266f052948329d14bfd287a3bc8e"
format_stamp: "Formatted at 2026-06-20 14:54:54 on dist-test-slave-3mqx"
server_key: "3896a5788302c099e73b52f00338af79"
server_key_iv: "d63084fffbb97b71925990fd236f1f57"
server_key_version: "encryptionkey@0"
I20260620 14:54:54.464380 31972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:54.470230 31972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:54.471170 31972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:54.471498 31972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:54.491151 31972 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:34685
I20260620 14:54:54.491204 32037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:34685 every 8 connection(s)
I20260620 14:54:54.492690 31972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:54.493163 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 31972
I20260620 14:54:54.493456 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:54.494282 22143 external_mini_cluster.cc:1428] Setting key 12bc8f52a928eab3cd1178da29128553
I20260620 14:54:54.497386 32038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:54.503520 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:54.496530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.21.159.254:34685)
0620 14:54:54.497635 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:54.497647 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:54.497697 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:54:54.499441 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:54.499469 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:54.499925 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:54.500309 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:54.501584 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:54.501592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:54.501606 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:54.501621 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:54.501719 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:54.501737 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:54.501926 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:54.502036 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:54:54.502385 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":381,"threads_started":1}
I20260620 14:54:54.510191 32038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e: Bootstrap starting.
I20260620 14:54:54.513662 32038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:54.514768 32038 log.cc:824] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:54.518316 32038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e: No bootstrap required, opened a new log
I20260620 14:54:54.526023 32038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072e266f052948329d14bfd287a3bc8e" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34685 } }
I20260620 14:54:54.526469 32038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:54.526582 32038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072e266f052948329d14bfd287a3bc8e, State: Initialized, Role: FOLLOWER
I20260620 14:54:54.527242 32038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072e266f052948329d14bfd287a3bc8e" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34685 } }
I20260620 14:54:54.527495 32038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:54.527637 32038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:54.527813 32038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:54.529969 32038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072e266f052948329d14bfd287a3bc8e" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34685 } }
I20260620 14:54:54.530503 32038 leader_election.cc:302] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 072e266f052948329d14bfd287a3bc8e; no voters:
I20260620 14:54:54.531250 32038 leader_election.cc:288] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:54.531500 32043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:54.532353 32043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [term 1 LEADER]: Becoming Leader. State: Replica: 072e266f052948329d14bfd287a3bc8e, State: Running, Role: LEADER
I20260620 14:54:54.532985 32043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072e266f052948329d14bfd287a3bc8e" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34685 } }
I20260620 14:54:54.533819 32038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:54.536482 32044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "072e266f052948329d14bfd287a3bc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072e266f052948329d14bfd287a3bc8e" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34685 } } }
I20260620 14:54:54.536676 32045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 072e266f052948329d14bfd287a3bc8e. Latest consensus state: current_term: 1 leader_uuid: "072e266f052948329d14bfd287a3bc8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072e266f052948329d14bfd287a3bc8e" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34685 } } }
I20260620 14:54:54.536966 32044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:54.537006 32045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:54.544234 32050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:54.550498 32050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:54.560678 32050 catalog_manager.cc:1347] Generated new cluster ID: ab1f07016d0b4372ad536bde5df20a59
I20260620 14:54:54.560813 32050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:54.581585 32050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:54.582074 32050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:54.592813 32050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 072e266f052948329d14bfd287a3bc8e: Generated new TSK 0
I20260620 14:54:54.593752 32050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:54.601842 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:34685
--builtin_ntp_servers=127.21.159.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:54:54.958436 32062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:54.958745 32062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:54.958902 32062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:54.966599 32062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:54.966715 32062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:54.966892 32062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:54.976785 32062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:34685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:54.978171 32062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:54.980122 32062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:54.993825 32067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:54.993964 32068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:54.993885 32070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:54.995021 32062 server_base.cc:1034] running on GCE node
I20260620 14:54:54.995780 32062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:54.996848 32062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:54.998116 32062 hybrid_clock.cc:648] HybridClock initialized: now 1781967294998041 us; error 61 us; skew 500 ppm
I20260620 14:54:54.998553 32062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:55.001147 32062 webserver.cc:460] Webserver started at http://127.21.159.193:35255/ using document root <none> and password file <none>
I20260620 14:54:55.001940 32062 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:55.002082 32062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:55.002461 32062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:55.007920 32062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "06aa7ddf46004313b5a56e980d14bfa2"
format_stamp: "Formatted at 2026-06-20 14:54:54 on dist-test-slave-3mqx"
server_key: "1ee11cf3d276845846a7a85545516289"
server_key_iv: "c15008d1ce5f2020c5c8580857224cf9"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.008807 32062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "06aa7ddf46004313b5a56e980d14bfa2"
format_stamp: "Formatted at 2026-06-20 14:54:54 on dist-test-slave-3mqx"
server_key: "1ee11cf3d276845846a7a85545516289"
server_key_iv: "c15008d1ce5f2020c5c8580857224cf9"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.014461 32062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260620 14:54:55.018481 32076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.020282 32062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:54:55.020545 32062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "06aa7ddf46004313b5a56e980d14bfa2"
format_stamp: "Formatted at 2026-06-20 14:54:54 on dist-test-slave-3mqx"
server_key: "1ee11cf3d276845846a7a85545516289"
server_key_iv: "c15008d1ce5f2020c5c8580857224cf9"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.020849 32062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:55.028329 32062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:55.029443 32062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:55.029840 32062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:55.031217 32062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:55.032995 32062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:55.033171 32062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.033334 32062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:55.033445 32062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.074550 32062 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:42893
I20260620 14:54:55.074605 32188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:42893 every 8 connection(s)
I20260620 14:54:55.076252 32062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:55.078835 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 32062
I20260620 14:54:55.079118 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:55.079962 22143 external_mini_cluster.cc:1428] Setting key 34cb36d9f85cae726c8d827f6f7b48a3
I20260620 14:54:55.082845 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:34685
--builtin_ntp_servers=127.21.159.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:54:55.085719 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.079266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:59709 (local address 127.21.159.254:34685)
0620 14:54:55.079783 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.079791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.081357 (+ 1566us) server_negotiation.cc:407] Connection header received
0620 14:54:55.083219 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.083226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.083292 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.083428 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:55.084922 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:55.084928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:55.084933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:55.084953 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:55.085004 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:55.085017 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:55.085166 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.085368 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.085439 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":234,"threads_started":1}
I20260620 14:54:55.086797 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.079931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:34685 (local address 127.21.159.193:59709)
0620 14:54:55.081148 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.081207 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.082799 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.083696 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.083706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.083821 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:55.084585 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:55.084587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:55.084630 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:55.084729 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:55.085124 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:55.085135 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:55.085145 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.085459 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.085758 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":366,"threads_started":1}
I20260620 14:54:55.088784 32189 heartbeater.cc:347] Connected to a master server at 127.21.159.254:34685
I20260620 14:54:55.089289 32189 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:55.089975 32189 heartbeater.cc:511] Master 127.21.159.254:34685 requested a full tablet report, sending...
I20260620 14:54:55.092795 32003 ts_manager.cc:194] Registered new tserver with Master: 06aa7ddf46004313b5a56e980d14bfa2 (127.21.159.193:42893)
W20260620 14:54:55.437340 32193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:55.437675 32193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:55.437851 32193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:55.445493 32193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:55.445640 32193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:55.445812 32193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:55.455538 32193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:34685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:55.457437 32193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:55.459692 32193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:55.473958 32199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.474057 32201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.474078 32198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.475487 32193 server_base.cc:1034] running on GCE node
I20260620 14:54:55.476310 32193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:55.477451 32193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:55.478749 32193 hybrid_clock.cc:648] HybridClock initialized: now 1781967295478670 us; error 81 us; skew 500 ppm
I20260620 14:54:55.479234 32193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:55.482038 32193 webserver.cc:460] Webserver started at http://127.21.159.194:40077/ using document root <none> and password file <none>
I20260620 14:54:55.482883 32193 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:55.483044 32193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:55.483487 32193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:55.486685 32193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "0d8737d3ffcc4cfd9c3eda0d3b96c273"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-3mqx"
server_key: "9ea0c02d4afe869175bed1561051ff8f"
server_key_iv: "5c4d2f2ff42e8163171f56e89cdf936f"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.487624 32193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "0d8737d3ffcc4cfd9c3eda0d3b96c273"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-3mqx"
server_key: "9ea0c02d4afe869175bed1561051ff8f"
server_key_iv: "5c4d2f2ff42e8163171f56e89cdf936f"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.493670 32193 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:54:55.497789 32207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.499523 32193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:55.499725 32193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "0d8737d3ffcc4cfd9c3eda0d3b96c273"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-3mqx"
server_key: "9ea0c02d4afe869175bed1561051ff8f"
server_key_iv: "5c4d2f2ff42e8163171f56e89cdf936f"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.500020 32193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:55.506400 32193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:55.507395 32193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:55.507709 32193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:55.508795 32193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:55.510437 32193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:55.510535 32193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.510723 32193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:55.510815 32193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.547401 32193 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:43517
I20260620 14:54:55.547443 32319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:43517 every 8 connection(s)
I20260620 14:54:55.548895 32193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:55.549556 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 32193
I20260620 14:54:55.549783 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:55.550505 22143 external_mini_cluster.cc:1428] Setting key b48aea0760d4acbb5f94fb7c3a7bd5a5
I20260620 14:54:55.553864 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:34685
--builtin_ntp_servers=127.21.159.212:41317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:54:55.558986 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.552191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:59305 (local address 127.21.159.254:34685)
0620 14:54:55.552440 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.552447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.554363 (+ 1916us) server_negotiation.cc:407] Connection header received
0620 14:54:55.556266 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.556276 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.556366 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.556493 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:55.558047 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:55.558056 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:55.558065 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:55.558092 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:55.558140 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:55.558162 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:55.558302 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.558517 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.558595 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 14:54:55.559877 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.552989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:34685 (local address 127.21.159.194:59305)
0620 14:54:55.554164 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.554219 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.555948 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.556728 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.556737 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.556841 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:55.557694 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:55.557697 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:55.557738 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:55.557852 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:55.558292 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:55.558301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:55.558308 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.558565 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.558780 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":372,"threads_started":1}
I20260620 14:54:55.561568 32320 heartbeater.cc:347] Connected to a master server at 127.21.159.254:34685
I20260620 14:54:55.561906 32320 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:55.562641 32320 heartbeater.cc:511] Master 127.21.159.254:34685 requested a full tablet report, sending...
I20260620 14:54:55.564345 32003 ts_manager.cc:194] Registered new tserver with Master: 0d8737d3ffcc4cfd9c3eda0d3b96c273 (127.21.159.194:43517)
W20260620 14:54:55.897506 32323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:55.897825 32323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:55.898005 32323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:55.905814 32323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:55.905932 32323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:55.906105 32323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:55.915755 32323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:41317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:34685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:55.917150 32323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:55.919148 32323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:55.933384 32328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.933354 32329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.933424 32331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.935026 32323 server_base.cc:1034] running on GCE node
I20260620 14:54:55.935750 32323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:55.936815 32323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:55.938047 32323 hybrid_clock.cc:648] HybridClock initialized: now 1781967295937996 us; error 39 us; skew 500 ppm
I20260620 14:54:55.938381 32323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:54:55.941045 32323 webserver.cc:460] Webserver started at http://127.21.159.195:38473/ using document root <none> and password file <none>
I20260620 14:54:55.941881 32323 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:55.941998 32323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:55.942320 32323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:55.945822 32323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "90abf12dbe3c4976b6376da8af70dddb"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-3mqx"
server_key: "3783b1b814abf2335ac072529cbb8de6"
server_key_iv: "fed42f79169e5f0dfe202f1f1b0149f8"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.946714 32323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "90abf12dbe3c4976b6376da8af70dddb"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-3mqx"
server_key: "3783b1b814abf2335ac072529cbb8de6"
server_key_iv: "fed42f79169e5f0dfe202f1f1b0149f8"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.953215 32323 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:54:55.957278 32337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.958963 32323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:54:55.959224 32323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "90abf12dbe3c4976b6376da8af70dddb"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-3mqx"
server_key: "3783b1b814abf2335ac072529cbb8de6"
server_key_iv: "fed42f79169e5f0dfe202f1f1b0149f8"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.959442 32323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:55.965538 32323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:55.966398 32323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:55.966648 32323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:55.967638 32323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:55.969221 32323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:55.969339 32323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.969430 32323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:55.969480 32323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.003746 32323 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:42971
I20260620 14:54:56.003872 32449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:42971 every 8 connection(s)
I20260620 14:54:56.005257 32323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:56.010891 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 32323
I20260620 14:54:56.011153 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:56.011739 22143 external_mini_cluster.cc:1428] Setting key 1da99b923e81d81970ea5878b691a7cc
I20260620 14:54:56.015638 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.008202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:43991 (local address 127.21.159.254:34685)
0620 14:54:56.008415 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.008422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.010065 (+ 1643us) server_negotiation.cc:407] Connection header received
0620 14:54:56.012702 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.012707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.012778 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.012863 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.014742 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.014747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.014752 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.014773 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.014813 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.014830 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.014956 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.015224 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.015301 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:54:56.016782 32452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.008836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:34685 (local address 127.21.159.195:43991)
0620 14:54:56.009890 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.009931 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.012322 (+ 2391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.013147 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.013156 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.013317 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:56.014333 (+ 1016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:56.014337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:56.014413 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:56.014535 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:56.014942 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:56.014950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:56.014957 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.015318 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.015700 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":333,"threads_started":1}
I20260620 14:54:56.018383 32450 heartbeater.cc:347] Connected to a master server at 127.21.159.254:34685
I20260620 14:54:56.018690 32450 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:56.019367 32450 heartbeater.cc:511] Master 127.21.159.254:34685 requested a full tablet report, sending...
I20260620 14:54:56.020939 32003 ts_manager.cc:194] Registered new tserver with Master: 90abf12dbe3c4976b6376da8af70dddb (127.21.159.195:42971)
I20260620 14:54:56.028646 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:54:56.035385 32191 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:56.033564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.21.159.254:34685)
0620 14:54:56.033805 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.033812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.034352 (+ 540us) server_negotiation.cc:407] Connection header received
0620 14:54:56.034511 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.034516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.034570 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.034707 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.035062 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49770: BlockingRecv error: recv got EOF from 127.0.0.1:49770 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:54:56.036274 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 32062
I20260620 14:54:56.040123 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.037245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.21.159.193:42893)
0620 14:54:56.037849 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.037856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.037882 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:54:56.037941 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.037947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.038173 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.038300 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.039385 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.039393 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.039405 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.039423 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.039462 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.039475 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.039580 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.039672 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.039750 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":302,"threads_started":1}
I20260620 14:54:56.041394 32184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:56.122455 32189 heartbeater.cc:503] Master 127.21.159.254:34685 was elected leader, sending a full tablet report...
I20260620 14:54:56.124166 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 32062
W20260620 14:54:56.141561 31987 connection.cc:535] server connection from 127.21.159.193:59709 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:54:56.141873 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 32193
I20260620 14:54:56.146185 32461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.143035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.21.159.194:43517)
0620 14:54:56.143624 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.143631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.143661 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:54:56.143725 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.143731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.144026 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.144252 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.145316 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.145327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.145350 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.145372 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.145422 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.145439 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.145593 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.145755 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.145866 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":309,"threads_started":1}
I20260620 14:54:56.147493 32315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:56.229511 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 32193
I20260620 14:54:56.246017 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 32323
I20260620 14:54:56.250332 32463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.247200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.21.159.195:42971)
0620 14:54:56.247829 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.247836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.247866 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:54:56.247928 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.247934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.248158 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.248347 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.249530 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.249538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.249550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.249567 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.249605 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.249618 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.249816 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.249928 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.250035 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":314,"threads_started":1}
I20260620 14:54:56.251811 32445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:56.335736 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 32323
I20260620 14:54:56.352149 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 31972
I20260620 14:54:56.353370 32033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:54:56.469127 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 31972
2026-06-20T14:54:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2485 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:56 dist-test-slave-3mqx krb5kdc[32468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:56 dist-test-slave-3mqx krb5kdc[32468](info): set up 2 sockets
Jun 20 14:54:56 dist-test-slave-3mqx krb5kdc[32468](info): commencing operation
krb5kdc: starting...
W20260620 14:54:57.600996 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:54:57 dist-test-slave-3mqx krb5kdc[32468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:54:57.847741 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36247
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:36247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:58.179981 32481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:58.180297 32481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:58.180467 32481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:58.187773 32481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:58.187891 32481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:58.188014 32481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:58.188079 32481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:58.188189 32481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:58.198307 32481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:36247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:58.199891 32481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:58.201242 32481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:58.213871 32486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.213835 32487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.213796 32489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:58.214747 32481 server_base.cc:1034] running on GCE node
I20260620 14:54:58.215668 32481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:58.217129 32481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:58.218360 32481 hybrid_clock.cc:648] HybridClock initialized: now 1781967298218319 us; error 62 us; skew 500 ppm
I20260620 14:54:58.218711 32481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:58 dist-test-slave-3mqx krb5kdc[32468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:58.226876 32481 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:54:58.228569 32481 webserver.cc:460] Webserver started at http://127.21.159.254:37105/ using document root <none> and password file <none>
I20260620 14:54:58.229367 32481 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:58.229516 32481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:58.229840 32481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:58.233073 32481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "ff5eec240f734ea5a07b099d98fa28fb"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-3mqx"
server_key: "cdb2bde8a0b956445e355b61f67ca321"
server_key_iv: "8d6ce62b1903b111eb2fa304a7479fc2"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.233985 32481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "ff5eec240f734ea5a07b099d98fa28fb"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-3mqx"
server_key: "cdb2bde8a0b956445e355b61f67ca321"
server_key_iv: "8d6ce62b1903b111eb2fa304a7479fc2"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.239535 32481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 14:54:58.243212 32496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.244822 32481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:54:58.245062 32481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "ff5eec240f734ea5a07b099d98fa28fb"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-3mqx"
server_key: "cdb2bde8a0b956445e355b61f67ca321"
server_key_iv: "8d6ce62b1903b111eb2fa304a7479fc2"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.245306 32481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:58.267377 32481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:58.272215 32481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:58.272588 32481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:58.292333 32481 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:36247
I20260620 14:54:58.292349 32547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:36247 every 8 connection(s)
I20260620 14:54:58.293867 32481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:58.297757 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:58.301990 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 32481
I20260620 14:54:58.302248 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:54:58.303078 22143 external_mini_cluster.cc:1428] Setting key e79897c28a937c6e741f714bdc56890b
I20260620 14:54:58.310065 32548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb: Bootstrap starting.
I20260620 14:54:58.313406 32548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:58.314488 32548 log.cc:824] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:58.317423 32548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb: No bootstrap required, opened a new log
Jun 20 14:54:58 dist-test-slave-3mqx krb5kdc[32468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:58.327251 32548 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5eec240f734ea5a07b099d98fa28fb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36247 } }
I20260620 14:54:58.327662 32548 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:58.327773 32548 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5eec240f734ea5a07b099d98fa28fb, State: Initialized, Role: FOLLOWER
I20260620 14:54:58.328332 32551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.305433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.21.159.254:36247)
0620 14:54:58.306309 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.306322 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.306363 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:54:58.308669 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.308695 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.309145 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.309561 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:58.311182 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.312553 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.314003 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.314770 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.314987 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.319490 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:58.319500 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:58.319517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:58.319562 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:58.323387 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.324365 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.324374 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.324382 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.324596 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.325041 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.325045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.325047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.325693 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:58.325994 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.326462 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.326961 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":427,"threads_started":1}
I20260620 14:54:58.328531 32548 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5eec240f734ea5a07b099d98fa28fb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36247 } }
I20260620 14:54:58.328828 32548 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:58.328989 32548 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:58.329208 32548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:58.331822 32548 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5eec240f734ea5a07b099d98fa28fb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36247 } }
I20260620 14:54:58.332468 32548 leader_election.cc:302] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff5eec240f734ea5a07b099d98fa28fb; no voters:
I20260620 14:54:58.333225 32548 leader_election.cc:288] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:58.333385 32553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:58.335671 32553 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [term 1 LEADER]: Becoming Leader. State: Replica: ff5eec240f734ea5a07b099d98fa28fb, State: Running, Role: LEADER
I20260620 14:54:58.335790 32548 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:58.336375 32553 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5eec240f734ea5a07b099d98fa28fb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36247 } }
I20260620 14:54:58.339529 32555 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff5eec240f734ea5a07b099d98fa28fb. Latest consensus state: current_term: 1 leader_uuid: "ff5eec240f734ea5a07b099d98fa28fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5eec240f734ea5a07b099d98fa28fb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36247 } } }
I20260620 14:54:58.339946 32555 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:58.339628 32554 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff5eec240f734ea5a07b099d98fa28fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5eec240f734ea5a07b099d98fa28fb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36247 } } }
I20260620 14:54:58.340073 32554 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:58.340878 32560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:58.347043 32560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:58.359206 32560 catalog_manager.cc:1347] Generated new cluster ID: a95f4e4f45cf468d94623625ced239ff
I20260620 14:54:58.359366 32560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:58.380648 32560 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:58.382032 32560 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:58.389741 32560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb: Generated new TSK 0
I20260620 14:54:58.390586 32560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:54:58.463730 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36247
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:58.805377 32574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:58.805686 32574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:58.805781 32574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:58.813465 32574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:58.813584 32574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:58.813699 32574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:54:58.823787 32574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:58.825119 32574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:58.826472 32574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:58.840678 32580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.840835 32582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.840950 32579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:58.841684 32574 server_base.cc:1034] running on GCE node
I20260620 14:54:58.842348 32574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:58.843358 32574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:58.844568 32574 hybrid_clock.cc:648] HybridClock initialized: now 1781967298844522 us; error 36 us; skew 500 ppm
I20260620 14:54:58.844863 32574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:58 dist-test-slave-3mqx krb5kdc[32468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:58.848249 32574 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:54:58.849967 32574 webserver.cc:460] Webserver started at http://127.21.159.193:44913/ using document root <none> and password file <none>
I20260620 14:54:58.850740 32574 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:58.850885 32574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:58.851275 32574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:58.854282 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "96428c496c8042828531c734cbd2f58a"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-3mqx"
server_key: "f96a23375a60af9b5dec9956a5500c08"
server_key_iv: "25ac0dc50e508e16e6db42b68ab59eba"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.855180 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "96428c496c8042828531c734cbd2f58a"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-3mqx"
server_key: "f96a23375a60af9b5dec9956a5500c08"
server_key_iv: "25ac0dc50e508e16e6db42b68ab59eba"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.861091 32574 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:54:58.864993 32589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.866475 32574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:54:58.866662 32574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "96428c496c8042828531c734cbd2f58a"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-3mqx"
server_key: "f96a23375a60af9b5dec9956a5500c08"
server_key_iv: "25ac0dc50e508e16e6db42b68ab59eba"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.866878 32574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:58.883189 32574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:58.887318 32574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:58.887620 32574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:58.888590 32574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:58.890134 32574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:58.890241 32574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.890384 32574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:58.890458 32574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.924250 32574 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:39023
I20260620 14:54:58.924293 32701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:39023 every 8 connection(s)
I20260620 14:54:58.925626 32574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:58.927826 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 32574
I20260620 14:54:58.928072 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:58.928869 22143 external_mini_cluster.cc:1428] Setting key d340091d704a85b177c6b37c8f7a2622
Jun 20 14:54:58 dist-test-slave-3mqx krb5kdc[32468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:58.951864 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.928822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:36169 (local address 127.21.159.254:36247)
0620 14:54:58.929383 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.929390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.930604 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 14:54:58.933260 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.933264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.933336 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.933432 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:58.935907 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.936931 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.939762 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.940444 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.940609 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.944992 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:58.944998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:58.945002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:58.945047 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:58.947853 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.949002 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.949006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.949009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.949143 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.949824 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.949833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.949835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.950177 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:58.950328 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.951389 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.951591 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":283,"threads_started":1}
I20260620 14:54:58.953675 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.929472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36247 (local address 127.21.159.193:36169)
0620 14:54:58.930423 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:58.930476 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:58.932878 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:58.933777 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:58.933793 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:58.934691 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:58.935658 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.935689 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.937148 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.937157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.939558 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.939580 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.940757 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.940764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.941143 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.941984 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:58.942007 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:58.944686 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:58.948048 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.948062 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.948085 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.948779 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.949315 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.949323 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.949331 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.949615 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.950320 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:58.950332 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:58.950894 (+ 562us) client_negotiation.cc:769] Sending connection context
0620 14:54:58.951286 (+ 392us) client_negotiation.cc:240] Negotiation successful
0620 14:54:58.951993 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":430,"threads_started":1}
I20260620 14:54:58.955621 32702 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36247
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:54:58.955996 32702 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:58.956914 32702 heartbeater.cc:511] Master 127.21.159.254:36247 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:54:58.959829 32513 ts_manager.cc:194] Registered new tserver with Master: 96428c496c8042828531c734cbd2f58a (127.21.159.193:39023)
I20260620 14:54:58.961341 32513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:36169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:54:58.987161 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36247
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:59.320940 32708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:59.321312 32708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:59.321491 32708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:59.329026 32708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:59.329198 32708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:59.329396 32708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:54:59.341136 32708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:59.342546 32708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:59.344019 32708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:59.357760 32713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.357846 32714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.357824 32716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.358929 32708 server_base.cc:1034] running on GCE node
I20260620 14:54:59.359655 32708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:59.360673 32708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:59.361878 32708 hybrid_clock.cc:648] HybridClock initialized: now 1781967299361800 us; error 62 us; skew 500 ppm
I20260620 14:54:59.362236 32708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:59 dist-test-slave-3mqx krb5kdc[32468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:59.365993 32708 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:54:59.367571 32708 webserver.cc:460] Webserver started at http://127.21.159.194:46547/ using document root <none> and password file <none>
I20260620 14:54:59.368294 32708 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:59.368446 32708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:59.368752 32708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:59.371840 32708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "013f7b7758664865814e232d224a6b22"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-3mqx"
server_key: "173f5b4361377510727ff86af3807833"
server_key_iv: "7dbffe9e0a3e24c3a8c0878e8980c87f"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.372678 32708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "013f7b7758664865814e232d224a6b22"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-3mqx"
server_key: "173f5b4361377510727ff86af3807833"
server_key_iv: "7dbffe9e0a3e24c3a8c0878e8980c87f"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.378171 32708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:54:59.382005 32723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.383586 32708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:59.383778 32708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "013f7b7758664865814e232d224a6b22"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-3mqx"
server_key: "173f5b4361377510727ff86af3807833"
server_key_iv: "7dbffe9e0a3e24c3a8c0878e8980c87f"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.383989 32708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:59.399207 32708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:59.403587 32708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:59.403908 32708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:59.404896 32708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:59.406265 32708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:59.406373 32708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.406533 32708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:59.406617 32708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.440557 32708 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:44161
I20260620 14:54:59.440616 367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:44161 every 8 connection(s)
I20260620 14:54:59.441908 32708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:59.450850 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 32708
I20260620 14:54:59.451159 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:59.451596 22143 external_mini_cluster.cc:1428] Setting key 3d1571694b1d5f3a5855d240d9aa5219
Jun 20 14:54:59 dist-test-slave-3mqx krb5kdc[32468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:54:59.463790 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.444473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:43209 (local address 127.21.159.254:36247)
0620 14:54:59.444684 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.444690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.445903 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 14:54:59.447730 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.447735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.447807 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.447900 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:59.449494 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.450334 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.452401 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.453008 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.453161 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.457250 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:59.457256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:59.457259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:59.457293 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:59.460154 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:59.460974 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:59.460982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:59.460986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:59.461160 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:59.461873 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:59.461898 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:59.461904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:59.462254 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:59.462369 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.463344 (+ 975us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.463525 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":75}
I20260620 14:54:59.465458 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.445054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36247 (local address 127.21.159.194:43209)
0620 14:54:59.445741 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.445783 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.447460 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.448140 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.448148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.448696 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:59.449318 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.449337 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.450549 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.450553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.452200 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.452219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.453251 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.453259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.453624 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.454416 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:59.454458 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:59.456985 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:59.460333 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:59.460343 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:59.460363 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:59.460820 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:59.461332 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:59.461344 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:59.461349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:59.461662 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:59.462388 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:59.462400 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:59.462906 (+ 506us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.463262 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.463895 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":286,"threads_started":1}
I20260620 14:54:59.467268 368 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36247
I20260620 14:54:59.467633 368 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:59.468588 368 heartbeater.cc:511] Master 127.21.159.254:36247 requested a full tablet report, sending...
I20260620 14:54:59.470412 32513 ts_manager.cc:194] Registered new tserver with Master: 013f7b7758664865814e232d224a6b22 (127.21.159.194:44161)
I20260620 14:54:59.471763 32513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:43209
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:54:59.507602 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36247
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:59.851912 373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:59.852216 373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:59.852310 373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:59.859877 373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:59.860070 373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:59.860270 373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:54:59.870283 373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:36247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:54:59.871690 373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:59.873016 373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:59.886641 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.886757 381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.886746 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.888047 373 server_base.cc:1034] running on GCE node
I20260620 14:54:59.888769 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:59.889818 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:59.891104 373 hybrid_clock.cc:648] HybridClock initialized: now 1781967299890986 us; error 55 us; skew 500 ppm
I20260620 14:54:59.891480 373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:54:59 dist-test-slave-3mqx krb5kdc[32468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:59.894690 373 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:54:59.896435 373 webserver.cc:460] Webserver started at http://127.21.159.195:37361/ using document root <none> and password file <none>
I20260620 14:54:59.897169 373 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:59.897317 373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:59.897644 373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:59.900736 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "78829bed555d4ea4b3e138379321d05c"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-3mqx"
server_key: "64e7b111ea7deeeb5eb75bc1514592ad"
server_key_iv: "292e0741e1b3acd353049b08c4e438b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.901557 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "78829bed555d4ea4b3e138379321d05c"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-3mqx"
server_key: "64e7b111ea7deeeb5eb75bc1514592ad"
server_key_iv: "292e0741e1b3acd353049b08c4e438b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.907126 373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 14:54:59.910892 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.912389 373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:54:59.912595 373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "78829bed555d4ea4b3e138379321d05c"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-3mqx"
server_key: "64e7b111ea7deeeb5eb75bc1514592ad"
server_key_iv: "292e0741e1b3acd353049b08c4e438b7"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.912817 373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:59.945817 373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:59.950683 373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:59.951020 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:59.952054 373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:59.953609 373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:59.953719 373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.953896 373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:59.954017 373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 14:54:59.964931 32702 heartbeater.cc:503] Master 127.21.159.254:36247 was elected leader, sending a full tablet report...
I20260620 14:54:59.990669 373 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:39713
I20260620 14:54:59.990710 500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:39713 every 8 connection(s)
I20260620 14:54:59.992449 373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:55:00.001592 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 373
I20260620 14:55:00.001799 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:55:00.002257 22143 external_mini_cluster.cc:1428] Setting key 4ecd9b3bc057c4c1749d71eb7b6fb887
Jun 20 14:55:00 dist-test-slave-3mqx krb5kdc[32468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967299, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:55:00.012619 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.995219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:52951 (local address 127.21.159.254:36247)
0620 14:54:59.995695 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.995699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.996640 (+ 941us) server_negotiation.cc:407] Connection header received
0620 14:54:59.998507 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.998512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.998608 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.998728 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:00.000332 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.001105 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.002879 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.003478 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.003599 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.007277 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:00.007283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:00.007286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:00.007318 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:00.009744 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.010436 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.010440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.010443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.010564 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.010926 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.010929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.010931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.011338 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:00.011491 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.012140 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.012320 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":271,"threads_started":1}
I20260620 14:55:00.013676 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.995748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36247 (local address 127.21.159.195:52951)
0620 14:54:59.996462 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.996499 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.998209 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.998976 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.998984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.999582 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:00.000157 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.000174 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.001287 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.001291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.002727 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.002739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.003682 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.003689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.003937 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.004625 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:00.004649 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:00.007026 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:00.009871 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:00.009878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:00.009892 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:00.010324 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:00.010653 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:00.010656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:00.010659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:00.010831 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:00.011454 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:00.011461 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:00.011806 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 14:55:00.012052 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:55:00.012462 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":296,"threads_started":1}
I20260620 14:55:00.015273 501 heartbeater.cc:347] Connected to a master server at 127.21.159.254:36247
I20260620 14:55:00.015632 501 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:00.016673 501 heartbeater.cc:511] Master 127.21.159.254:36247 requested a full tablet report, sending...
I20260620 14:55:00.018375 32513 ts_manager.cc:194] Registered new tserver with Master: 78829bed555d4ea4b3e138379321d05c (127.21.159.195:39713)
I20260620 14:55:00.019232 32513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:52951
I20260620 14:55:00.030483 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:55:00.048362 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.035517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.21.159.254:36247)
0620 14:55:00.035739 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.035744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.036320 (+ 576us) server_negotiation.cc:407] Connection header received
0620 14:55:00.036554 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.036559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.036625 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.036729 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:00.038207 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.038940 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.040218 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.040621 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.040762 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.043043 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:00.043090 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:00.043097 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:00.043138 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:00.045550 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.046261 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.046266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.046270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.046397 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.046785 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.046792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.046795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.047204 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:00.047451 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.047781 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.048023 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 14:55:00.052829 32513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:55:00.060046 32513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:55:00.088263 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.073521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.21.159.195:39713)
0620 14:55:00.075409 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.075422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.075462 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:55:00.075587 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.075599 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.075941 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.076196 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:00.077717 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.079930 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.083720 (+ 3790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.085889 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.086113 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.086365 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.087398 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.087672 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1654,"thread_start_us":1745,"threads_started":1}
I20260620 14:55:00.089403 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.072956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:39713 (local address 127.0.0.1:60766)
0620 14:55:00.074247 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:00.074265 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:00.074373 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:00.076942 (+ 2569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:00.076950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:00.076991 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:00.077435 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.077464 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.080171 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.080179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.083463 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.083483 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.086613 (+ 3130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.086622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.087004 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.087106 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 14:55:00.088581 (+ 1475us) client_negotiation.cc:240] Negotiation successful
0620 14:55:00.088887 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":975,"threads_started":1}
I20260620 14:55:00.090734 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.072770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:39023 (local address 127.0.0.1:51160)
0620 14:55:00.073548 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:00.073604 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:00.073835 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:00.077571 (+ 3736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:00.077578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:00.077613 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:00.079008 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.079029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.082752 (+ 3723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.082766 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.087403 (+ 4637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.087423 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.089997 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.090003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.090194 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.090226 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:55:00.090292 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:55:00.090422 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":330,"threads_started":1}
I20260620 14:55:00.091874 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.073074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.21.159.193:39023)
0620 14:55:00.076942 (+ 3868us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.076960 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.076988 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:55:00.077080 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.077089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.077325 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.077768 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:00.079323 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.081097 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.087698 (+ 6601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.089697 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.091096 (+ 1399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.091283 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.091353 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.091539 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3713,"thread_start_us":3558,"threads_started":1}
I20260620 14:55:00.094635 436 tablet_service.cc:1467] Processing CreateTablet for tablet 670d602f27a945399453dd94f599bf8a (DEFAULT_TABLE table=test-table [id=f4cebc2f769644deb20dcfba32f0e64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:00.095517 32637 tablet_service.cc:1467] Processing CreateTablet for tablet 670d602f27a945399453dd94f599bf8a (DEFAULT_TABLE table=test-table [id=f4cebc2f769644deb20dcfba32f0e64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:00.096989 32637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670d602f27a945399453dd94f599bf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:00.096988 436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670d602f27a945399453dd94f599bf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:00.098577 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.075092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:44161 (local address 127.0.0.1:33096)
0620 14:55:00.090218 (+ 15126us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:00.090244 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:00.090376 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:00.091620 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:00.091628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:00.091661 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:00.091838 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.091852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.093935 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.093940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.095826 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.095843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.097801 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.097812 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.098022 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.098059 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:55:00.098135 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:55:00.098291 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14975,"threads_started":1}
I20260620 14:55:00.099596 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.075296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.21.159.194:44161)
0620 14:55:00.090937 (+ 15641us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.090946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.090973 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:55:00.091105 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.091114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.091389 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.092025 (+ 636us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:00.092250 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.093744 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.096051 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.097599 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.098856 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.099042 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.099142 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.099314 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15484,"thread_start_us":15386,"threads_started":1}
I20260620 14:55:00.104351 303 tablet_service.cc:1467] Processing CreateTablet for tablet 670d602f27a945399453dd94f599bf8a (DEFAULT_TABLE table=test-table [id=f4cebc2f769644deb20dcfba32f0e64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:00.106701 303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 670d602f27a945399453dd94f599bf8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:00.109520 520 tablet_bootstrap.cc:492] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Bootstrap starting.
I20260620 14:55:00.111936 521 tablet_bootstrap.cc:492] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Bootstrap starting.
I20260620 14:55:00.113161 520 tablet_bootstrap.cc:654] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:00.115103 520 log.cc:824] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:00.115721 521 tablet_bootstrap.cc:654] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:00.117159 521 log.cc:824] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:00.118535 520 tablet_bootstrap.cc:492] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: No bootstrap required, opened a new log
I20260620 14:55:00.118834 520 ts_tablet_manager.cc:1397] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 14:55:00.120390 521 tablet_bootstrap.cc:492] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: No bootstrap required, opened a new log
I20260620 14:55:00.120653 521 ts_tablet_manager.cc:1397] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 14:55:00.121224 524 tablet_bootstrap.cc:492] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Bootstrap starting.
I20260620 14:55:00.124462 524 tablet_bootstrap.cc:654] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:00.126637 524 log.cc:824] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:00.129578 524 tablet_bootstrap.cc:492] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: No bootstrap required, opened a new log
I20260620 14:55:00.129889 524 ts_tablet_manager.cc:1397] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20260620 14:55:00.129879 520 raft_consensus.cc:348] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.130368 520 raft_consensus.cc:374] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:00.130494 520 raft_consensus.cc:729] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96428c496c8042828531c734cbd2f58a, State: Initialized, Role: FOLLOWER
I20260620 14:55:00.131212 520 consensus_queue.cc:260] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.131577 521 raft_consensus.cc:348] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.132140 521 raft_consensus.cc:374] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:00.132280 521 raft_consensus.cc:729] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78829bed555d4ea4b3e138379321d05c, State: Initialized, Role: FOLLOWER
I20260620 14:55:00.132915 521 consensus_queue.cc:260] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.134065 501 heartbeater.cc:503] Master 127.21.159.254:36247 was elected leader, sending a full tablet report...
I20260620 14:55:00.134606 521 ts_tablet_manager.cc:1428] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260620 14:55:00.134588 520 ts_tablet_manager.cc:1428] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260620 14:55:00.141523 524 raft_consensus.cc:348] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.141955 524 raft_consensus.cc:374] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:00.142061 524 raft_consensus.cc:729] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013f7b7758664865814e232d224a6b22, State: Initialized, Role: FOLLOWER
I20260620 14:55:00.142758 524 consensus_queue.cc:260] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.143925 368 heartbeater.cc:503] Master 127.21.159.254:36247 was elected leader, sending a full tablet report...
I20260620 14:55:00.144584 524 ts_tablet_manager.cc:1428] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260620 14:55:00.159245 527 raft_consensus.cc:482] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:55:00.159530 527 raft_consensus.cc:504] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.161434 527 leader_election.cc:288] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96428c496c8042828531c734cbd2f58a (127.21.159.193:39023), 013f7b7758664865814e232d224a6b22 (127.21.159.194:44161)
I20260620 14:55:00.168530 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.162183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:55485 (local address 127.21.159.194:44161)
0620 14:55:00.162348 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.162353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.162370 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:55:00.162430 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.162434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.162497 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.162574 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:00.163026 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.164056 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.166436 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.167639 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.167831 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.167908 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.168127 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.168261 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:55:00.168550 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.161752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:44161 (local address 127.21.159.195:55485)
0620 14:55:00.162102 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:00.162123 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:00.162273 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:00.162626 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:00.162634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:00.162660 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:00.162878 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.162886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.164192 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.164196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.166260 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.166276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.167808 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.167812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.167958 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.167982 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:55:00.168059 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:55:00.168230 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260620 14:55:00.169004 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.161870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:55991 (local address 127.21.159.193:39023)
0620 14:55:00.162088 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.162094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.162684 (+ 590us) server_negotiation.cc:407] Connection header received
0620 14:55:00.162820 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.162824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.162900 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.163024 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:00.163531 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.164454 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.166707 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.168014 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.168188 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.168255 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.168647 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.168792 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:55:00.169545 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.161853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:39023 (local address 127.21.159.195:55991)
0620 14:55:00.162536 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:00.162576 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:00.162681 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:00.163042 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:00.163045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:00.163111 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:00.163374 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.163381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.164594 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.164604 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.166567 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:00.166576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.168230 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.168235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:00.168404 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.168445 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:55:00.169205 (+ 760us) client_negotiation.cc:240] Negotiation successful
0620 14:55:00.169305 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":827,"threads_started":1}
I20260620 14:55:00.169615 323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670d602f27a945399453dd94f599bf8a" candidate_uuid: "78829bed555d4ea4b3e138379321d05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013f7b7758664865814e232d224a6b22" is_pre_election: true
I20260620 14:55:00.170305 323 raft_consensus.cc:2454] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78829bed555d4ea4b3e138379321d05c in term 0.
I20260620 14:55:00.170662 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670d602f27a945399453dd94f599bf8a" candidate_uuid: "78829bed555d4ea4b3e138379321d05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96428c496c8042828531c734cbd2f58a" is_pre_election: true
I20260620 14:55:00.171144 391 leader_election.cc:302] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013f7b7758664865814e232d224a6b22, 78829bed555d4ea4b3e138379321d05c; no voters:
I20260620 14:55:00.171247 32657 raft_consensus.cc:2454] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78829bed555d4ea4b3e138379321d05c in term 0.
I20260620 14:55:00.171607 527 raft_consensus.cc:2781] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:55:00.171795 527 raft_consensus.cc:482] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:55:00.171929 527 raft_consensus.cc:3037] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:00.174188 527 raft_consensus.cc:504] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.174980 527 leader_election.cc:288] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [CANDIDATE]: Term 1 election: Requested vote from peers 96428c496c8042828531c734cbd2f58a (127.21.159.193:39023), 013f7b7758664865814e232d224a6b22 (127.21.159.194:44161)
I20260620 14:55:00.175518 323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670d602f27a945399453dd94f599bf8a" candidate_uuid: "78829bed555d4ea4b3e138379321d05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013f7b7758664865814e232d224a6b22"
I20260620 14:55:00.175557 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "670d602f27a945399453dd94f599bf8a" candidate_uuid: "78829bed555d4ea4b3e138379321d05c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96428c496c8042828531c734cbd2f58a"
I20260620 14:55:00.175750 323 raft_consensus.cc:3037] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:00.175829 32657 raft_consensus.cc:3037] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:00.178086 323 raft_consensus.cc:2454] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78829bed555d4ea4b3e138379321d05c in term 1.
W20260620 14:55:00.178131 32703 tablet.cc:2369] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:55:00.178203 32657 raft_consensus.cc:2454] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78829bed555d4ea4b3e138379321d05c in term 1.
I20260620 14:55:00.178608 391 leader_election.cc:302] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013f7b7758664865814e232d224a6b22, 78829bed555d4ea4b3e138379321d05c; no voters:
I20260620 14:55:00.178956 527 raft_consensus.cc:2781] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:00.179472 527 raft_consensus.cc:686] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 1 LEADER]: Becoming Leader. State: Replica: 78829bed555d4ea4b3e138379321d05c, State: Running, Role: LEADER
I20260620 14:55:00.179945 527 consensus_queue.cc:237] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } }
I20260620 14:55:00.184507 32513 catalog_manager.cc:5495] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c reported cstate change: term changed from 0 to 1, leader changed from <none> to 78829bed555d4ea4b3e138379321d05c (127.21.159.195). New cstate: current_term: 1 leader_uuid: "78829bed555d4ea4b3e138379321d05c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78829bed555d4ea4b3e138379321d05c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 39713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 } health_report { overall_health: UNKNOWN } } }
W20260620 14:55:00.194151 369 tablet.cc:2369] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:55:00.212123 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.206480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.21.159.195:39713)
0620 14:55:00.206724 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.206731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.206808 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:55:00.207117 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.207121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.207196 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.207360 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:00.207757 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.208482 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.209826 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.210232 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.210370 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.210910 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:00.211533 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:00.211691 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.211774 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.211859 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:55:00.220577 32657 raft_consensus.cc:1261] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 1 FOLLOWER]: Refusing update from remote peer 78829bed555d4ea4b3e138379321d05c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:00.220579 323 raft_consensus.cc:1261] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 1 FOLLOWER]: Refusing update from remote peer 78829bed555d4ea4b3e138379321d05c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:00.221624 527 consensus_queue.cc:1035] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96428c496c8042828531c734cbd2f58a" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 39023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:00.221983 530 consensus_queue.cc:1035] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [LEADER]: Connected to new peer: Peer: permanent_uuid: "013f7b7758664865814e232d224a6b22" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:00.251348 32513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:00.251653 32513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:00.255366 32513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff5eec240f734ea5a07b099d98fa28fb: Sending DeleteTablet for 3 replicas of tablet 670d602f27a945399453dd94f599bf8a
I20260620 14:55:00.256628 32637 tablet_service.cc:1514] Processing DeleteTablet for tablet 670d602f27a945399453dd94f599bf8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:00 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:51160
I20260620 14:55:00.256939 436 tablet_service.cc:1514] Processing DeleteTablet for tablet 670d602f27a945399453dd94f599bf8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:00 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:60766
I20260620 14:55:00.257655 543 tablet_replica.cc:330] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: stopping tablet replica
I20260620 14:55:00.257906 303 tablet_service.cc:1514] Processing DeleteTablet for tablet 670d602f27a945399453dd94f599bf8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:00 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:33096
I20260620 14:55:00.258188 543 raft_consensus.cc:2229] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:55:00.258575 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 32574
I20260620 14:55:00.258709 543 raft_consensus.cc:2258] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:00.262529 543 ts_tablet_manager.cc:1905] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:00.269886 543 ts_tablet_manager.cc:1918] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:00.270104 543 log.cc:1196] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/670d602f27a945399453dd94f599bf8a
I20260620 14:55:00.270779 545 tablet_replica.cc:330] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: stopping tablet replica
I20260620 14:55:00.271377 545 raft_consensus.cc:2229] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:55:00.271948 545 raft_consensus.cc:2258] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:00.272755 544 tablet_replica.cc:330] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: stopping tablet replica
I20260620 14:55:00.273201 544 raft_consensus.cc:2229] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:55:00.273862 544 raft_consensus.cc:2258] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:00.274638 545 ts_tablet_manager.cc:1905] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:00.274668 543 ts_tablet_manager.cc:1939] T 670d602f27a945399453dd94f599bf8a P 96428c496c8042828531c734cbd2f58a: Deleting consensus metadata
I20260620 14:55:00.276655 32498 catalog_manager.cc:4841] TS 96428c496c8042828531c734cbd2f58a (127.21.159.193:39023): tablet 670d602f27a945399453dd94f599bf8a (table test-table [id=f4cebc2f769644deb20dcfba32f0e64b]) successfully deleted
I20260620 14:55:00.277998 544 ts_tablet_manager.cc:1905] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:00.282171 545 ts_tablet_manager.cc:1918] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:00.282460 545 log.cc:1196] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/670d602f27a945399453dd94f599bf8a
I20260620 14:55:00.283099 545 ts_tablet_manager.cc:1939] T 670d602f27a945399453dd94f599bf8a P 013f7b7758664865814e232d224a6b22: Deleting consensus metadata
I20260620 14:55:00.285458 32499 catalog_manager.cc:4841] TS 013f7b7758664865814e232d224a6b22 (127.21.159.194:44161): tablet 670d602f27a945399453dd94f599bf8a (table test-table [id=f4cebc2f769644deb20dcfba32f0e64b]) successfully deleted
I20260620 14:55:00.285698 544 ts_tablet_manager.cc:1918] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:00.285952 544 log.cc:1196] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/670d602f27a945399453dd94f599bf8a
I20260620 14:55:00.286454 544 ts_tablet_manager.cc:1939] T 670d602f27a945399453dd94f599bf8a P 78829bed555d4ea4b3e138379321d05c: Deleting consensus metadata
Jun 20 14:55:00 dist-test-slave-3mqx krb5kdc[32468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:55:00.288224 32500 catalog_manager.cc:4841] TS 78829bed555d4ea4b3e138379321d05c (127.21.159.195:39713): tablet 670d602f27a945399453dd94f599bf8a (table test-table [id=f4cebc2f769644deb20dcfba32f0e64b]) successfully deleted
I20260620 14:55:00.296998 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.260079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.21.159.193:39023)
0620 14:55:00.260282 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.260290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.271163 (+ 10873us) server_negotiation.cc:407] Connection header received
0620 14:55:00.271892 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.271902 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.272020 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.272099 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:00.274023 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.275409 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.283128 (+ 7719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.283799 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.283917 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.289389 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:00.289409 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:00.289428 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:00.289477 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:00.293438 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.294434 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.294446 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.294450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.294630 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.295152 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.295158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.295161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.295647 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:00.295963 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.296295 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.296577 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:55:00.298161 32697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:00.397924 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 32574
I20260620 14:55:00.423413 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 32708
Jun 20 14:55:00 dist-test-slave-3mqx krb5kdc[32468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:55:00.441939 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.424286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.21.159.194:44161)
0620 14:55:00.424455 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.424459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.424479 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:55:00.424695 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.424699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.424775 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.424857 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:00.426429 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.427219 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.429101 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.429481 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.429601 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.434958 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:00.434974 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:00.434991 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:00.435032 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:00.438592 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.439524 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.439531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.439534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.439662 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.440228 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.440233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.440235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.440644 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:00.440942 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.441272 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.441526 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:55:00.442658 363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:00.532569 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 32708
I20260620 14:55:00.551301 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 373
Jun 20 14:55:00 dist-test-slave-3mqx krb5kdc[32468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:55:00.570844 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:00.552146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.21.159.195:39713)
0620 14:55:00.552323 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:00.552328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:00.552371 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:55:00.552598 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:00.552602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:00.552664 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:00.552749 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:00.554598 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.555543 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.557464 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:00.558057 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:00.558158 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:00.564157 (+ 5999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:00.564170 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:00.564186 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:00.564232 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:00.567628 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.568612 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.568619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.568621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.568794 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:00.569241 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:00.569245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:00.569246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:00.569635 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:00.569870 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:00.570270 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 14:55:00.570466 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:55:00.571704 496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:00.693778 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 373
I20260620 14:55:00.713385 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 32481
I20260620 14:55:00.714560 32543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:00.822366 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 32481
2026-06-20T14:55:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4353 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:00 dist-test-slave-3mqx krb5kdc[553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:00 dist-test-slave-3mqx krb5kdc[553](info): set up 2 sockets
Jun 20 14:55:00 dist-test-slave-3mqx krb5kdc[553](info): commencing operation
krb5kdc: starting...
W20260620 14:55:01.980103 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:55:02 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/rangeradmin_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/rangeradmin_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/rangerlookup_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/rangerlookup_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
I20260620 14:55:02.323611 22143 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmvfD4B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/postgres -l logfile start
2026-06-20 14:55:05.012 UTC [583] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:55:05.012 UTC [583] LOG: listening on IPv4 address "127.21.159.212", port 52387
2026-06-20 14:55:05.017 UTC [583] LOG: listening on Unix socket "/tmp/.s.PGSQL.52387"
2026-06-20 14:55:05.040 UTC [586] LOG: database system was shut down at 2026-06-20 14:55:03 UTC
2026-06-20 14:55:05.044 UTC [583] LOG: database system is ready to accept connections
I20260620 14:55:06.088302 22143 mini_postgres.cc:96] Postgres bound to 52387
2026-06-20 14:55:06.107 UTC [594] FATAL: database "slave" does not exist
127.21.159.212:52387 - accepting connections
I20260620 14:55:06.108240 22143 mini_ranger.cc:161] Starting Ranger...
I20260620 14:55:06.137056 22143 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:55:06.674245 22143 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:55:06.674357 22143 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-admin
2026-06-20 14:55:06,966 [I] DB FLAVOR :POSTGRES
2026-06-20 14:55:06,967 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:55:06,967 [I] Checking connection..
2026-06-20 14:55:06,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:55:07,227 [I] Checking connection passed.
2026-06-20 14:55:07,227 [I] --------- Verifying version history table ---------
2026-06-20 14:55:07,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:55:07,500 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:55:07,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:55:07,782 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:55:07,782 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:55:07,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:55:08.021 UTC [658] WARNING: there is no transaction in progress
2026-06-20 14:55:08,028 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:55:08,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:55:08,287 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:55:08,287 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:55:08,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:55:08,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:55:08,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3mqx.c.gcp-upstream.internal','N') ;"
2026-06-20 14:55:09,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:55:09,331 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:55:09,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:55:09,598 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:55:09,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:55:09,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:55:10,095 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:55:10,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:55:11.147 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.167 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.186 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.204 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.226 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.250 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.321 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.329 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.346 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.358 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.369 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.386 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.397 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.405 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.412 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:11.419 UTC [766] WARNING: there is no transaction in progress
2026-06-20 14:55:12,155 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:55:12,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:55:12,390 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:55:12,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:55:12,634 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:55:12,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:55:12,889 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:55:12,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:55:13,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:55:13,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3mqx.c.gcp-upstream.internal';"
2026-06-20 14:55:13,668 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:55:13,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:55:13,937 [I] DB_PATCHES have already been applied
I20260620 14:55:13.943742 22143 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmvfD4B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:55:13.943835 22143 mini_ranger.cc:193] Using host: 127.21.159.212
I20260620 14:55:13.960206 22143 mini_ranger.cc:227] Ranger admin URL: http://127.21.159.212:36715
Jun 20, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:55:15.119706 22143 mini_ranger.cc:160] Time spent starting Ranger: real 9.012s user 0.007s sys 0.033s
Jun 20, 2026 2:55:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:55:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Jun 20, 2026 2:55:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:55:37 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:55:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:55:38.563477 22143 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/rangerkms_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/rangerkms_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.159.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.159.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:55:38 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:55:38.739818 22143 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:55:38.762647 22143 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:55:39.769249 22143 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:55:39.769418 22143 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms
I20260620 14:55:39.769559 22143 mini_ranger_kms.cc:227] Using postgres at 127.21.159.212:52387
2026-06-20 14:55:40,202 [I] DB FLAVOR :POSTGRES
2026-06-20 14:55:40,203 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:55:40,203 [I] Checking connection
2026-06-20 14:55:40,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:55:40,475 [I] connection success
2026-06-20 14:55:40,475 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:55:40,475 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:55:40,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:55:40,748 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:55:40,748 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:55:40,748 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:55:40,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52387/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:55:41,063 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:55:41.547721 22143 mini_ranger_kms.cc:310] Created kms service
I20260620 14:55:41.753036 22143 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:55:41.806391 22143 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:55:41 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967338, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:55:42.256820 22143 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:55:42.257151 22143 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:55:42.257232 22143 mini_ranger_kms.cc:240] Using host: 127.21.159.212
I20260620 14:55:42.276072 22143 mini_ranger_kms.cc:276] Ranger KMS PID: 947
I20260620 14:55:42.276291 22143 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.159.212:58155
Jun 20, 2026 2:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:55:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:55:44.672046 22143 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.932s user 0.121s sys 0.322s
I20260620 14:55:44.672317 22143 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:55:44.672561 22143 mini_ranger_kms.cc:401] 127.21.159.212:58155/kms/v1/keys
Jun 20 14:55:44 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.159.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:55:45 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.159.212@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
Jun 20 14:55:45 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967344, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.159.212@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
Jun 20 14:55:47 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:55:47.867553 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38403
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:38403
--ranger_config_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:58155/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:48.216603 997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:48.216904 997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:48.217073 997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:48.224860 997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:48.224987 997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:48.225067 997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:48.225160 997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:48.225229 997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:48.235585 997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.159.212:58155/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:38403
--ranger_config_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:55:48.236976 997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:48.238432 997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:48.252425 1005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:48.252607 1003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:48.252465 1002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:48.253075 997 server_base.cc:1034] running on GCE node
I20260620 14:55:48.254139 997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:48.255837 997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:48.257073 997 hybrid_clock.cc:648] HybridClock initialized: now 1781967348257018 us; error 46 us; skew 500 ppm
I20260620 14:55:48.257414 997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:55:48 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:48.265014 997 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:55:48.267158 997 webserver.cc:460] Webserver started at http://127.21.159.254:39285/ using document root <none> and password file <none>
I20260620 14:55:48.267910 997 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:48.268013 997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:48.268290 997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:55:48 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967348, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:55:48.332659 997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "b412093d15ea4fd0a0ab86c007f4d42f"
format_stamp: "Formatted at 2026-06-20 14:55:48 on dist-test-slave-3mqx"
server_key: "3ad52cfb4fa93715419146e35ab32063"
server_key_iv: "e55271ee60ebaf1676c4432799b56893"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:48.333627 997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "b412093d15ea4fd0a0ab86c007f4d42f"
format_stamp: "Formatted at 2026-06-20 14:55:48 on dist-test-slave-3mqx"
server_key: "3ad52cfb4fa93715419146e35ab32063"
server_key_iv: "e55271ee60ebaf1676c4432799b56893"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:48.339936 997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 14:55:48.358635 1012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:48.360551 997 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 14:55:48.360778 997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "b412093d15ea4fd0a0ab86c007f4d42f"
format_stamp: "Formatted at 2026-06-20 14:55:48 on dist-test-slave-3mqx"
server_key: "3ad52cfb4fa93715419146e35ab32063"
server_key_iv: "e55271ee60ebaf1676c4432799b56893"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:48.361047 997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:48.382944 997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:48.387806 997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:48.388157 997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:48.416225 997 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:38403
I20260620 14:55:48.416244 1063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:38403 every 8 connection(s)
I20260620 14:55:48.417771 997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:48.418452 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 997
I20260620 14:55:48.418692 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:55:48.419541 22143 external_mini_cluster.cc:1428] Setting key 10ff06d165831d3f6bbb6cc970990a49
I20260620 14:55:48.422399 1064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:55:48 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:55:48.435201 1064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f: Bootstrap starting.
I20260620 14:55:48.438819 1064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:48.440016 1064 log.cc:824] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:48.442826 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:48.421655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.21.159.254:38403)
0620 14:55:48.422424 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:48.422435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:48.422516 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:55:48.424224 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:48.424246 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:48.424637 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:48.425048 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:48.426383 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:48.427872 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:48.429118 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:48.429804 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:48.429995 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:48.434645 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:48.434654 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:48.434671 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:48.434716 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:48.438088 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:48.438864 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:48.438902 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:48.438913 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:48.439176 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:48.439598 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:48.439604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:48.439608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:48.440239 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:48.440555 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:48.441444 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 14:55:48.441800 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":342,"threads_started":1}
I20260620 14:55:48.443720 1064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f: No bootstrap required, opened a new log
I20260620 14:55:48.452734 1064 raft_consensus.cc:348] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38403 } }
I20260620 14:55:48.453227 1064 raft_consensus.cc:374] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:48.453351 1064 raft_consensus.cc:729] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b412093d15ea4fd0a0ab86c007f4d42f, State: Initialized, Role: FOLLOWER
I20260620 14:55:48.453961 1064 consensus_queue.cc:260] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38403 } }
I20260620 14:55:48.454221 1064 raft_consensus.cc:388] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:48.454383 1064 raft_consensus.cc:482] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:48.454579 1064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:48.457016 1064 raft_consensus.cc:504] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38403 } }
I20260620 14:55:48.457608 1064 leader_election.cc:302] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b412093d15ea4fd0a0ab86c007f4d42f; no voters:
I20260620 14:55:48.458266 1064 leader_election.cc:288] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:48.458364 1069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:48.459681 1069 raft_consensus.cc:686] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [term 1 LEADER]: Becoming Leader. State: Replica: b412093d15ea4fd0a0ab86c007f4d42f, State: Running, Role: LEADER
I20260620 14:55:48.460222 1069 consensus_queue.cc:237] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38403 } }
I20260620 14:55:48.461040 1064 sys_catalog.cc:564] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:48.463191 1071 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b412093d15ea4fd0a0ab86c007f4d42f. Latest consensus state: current_term: 1 leader_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38403 } } }
I20260620 14:55:48.463675 1071 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:48.464134 1070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b412093d15ea4fd0a0ab86c007f4d42f" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38403 } } }
I20260620 14:55:48.464529 1070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:48.467187 1077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:48.473114 1077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:48.474241 1064 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:55:48.491390 1077 catalog_manager.cc:1347] Generated new cluster ID: 863f618df89d4a40a57f554ece2a964b
I20260620 14:55:48.491652 1077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:48.526058 1077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:48.528427 1077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:48.543025 1077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f: Generated new TSK 0
I20260620 14:55:48.544531 1077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:55:49.350662 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:58155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38403
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:55:49 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:55:49.805083 1109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:49.805529 1109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:49.805696 1109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:49.818009 1109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:49.818185 1109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:49.818490 1109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:55:49.834960 1109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.159.212:58155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:55:49.836884 1109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:49.838784 1109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:49.856235 1120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:49.856628 1119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:49.857292 1109 server_base.cc:1034] running on GCE node
W20260620 14:55:49.857646 1122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:49.858367 1109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:49.859534 1109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:49.860785 1109 hybrid_clock.cc:648] HybridClock initialized: now 1781967349860730 us; error 39 us; skew 500 ppm
I20260620 14:55:49.861130 1109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:55:49 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:49.865093 1109 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:55:49.878860 1109 webserver.cc:460] Webserver started at http://127.21.159.193:46053/ using document root <none> and password file <none>
I20260620 14:55:49.880062 1109 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:49.880237 1109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:49.880599 1109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:55:49 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:55:49.901439 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "b8160d6854034864af84586c00392952"
format_stamp: "Formatted at 2026-06-20 14:55:49 on dist-test-slave-3mqx"
server_key: "52c28b08f852044b2903eb601bde64fe"
server_key_iv: "7613353ceb601f93e12112950dc23972"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:49.902575 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "b8160d6854034864af84586c00392952"
format_stamp: "Formatted at 2026-06-20 14:55:49 on dist-test-slave-3mqx"
server_key: "52c28b08f852044b2903eb601bde64fe"
server_key_iv: "7613353ceb601f93e12112950dc23972"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:49.925261 1109 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.008s sys 0.012s
I20260620 14:55:49.962154 1129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:49.966375 1109 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.011s sys 0.009s
I20260620 14:55:49.966840 1109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "b8160d6854034864af84586c00392952"
format_stamp: "Formatted at 2026-06-20 14:55:49 on dist-test-slave-3mqx"
server_key: "52c28b08f852044b2903eb601bde64fe"
server_key_iv: "7613353ceb601f93e12112950dc23972"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:49.967284 1109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:50.010686 1109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:50.028383 1109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:50.028815 1109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:50.043591 1109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:50.056087 1109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:50.056332 1109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 14:55:50.056550 1109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:50.056663 1109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:55:50 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:55:50.388957 1109 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:33969
I20260620 14:55:50.388998 1241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:33969 every 8 connection(s)
I20260620 14:55:50.390867 1109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:55:50.392014 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 1109
I20260620 14:55:50.392395 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:55:50.393564 22143 external_mini_cluster.cc:1428] Setting key 78e8a122d2782e610329c14a31f44ed4
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Jun 20 14:55:50 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:55:50.478148 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:50.415789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:59781 (local address 127.21.159.254:38403)
0620 14:55:50.440388 (+ 24599us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:50.440395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:50.443127 (+ 2732us) server_negotiation.cc:407] Connection header received
0620 14:55:50.449785 (+ 6658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:50.449794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:50.449913 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:50.450021 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:50.456819 (+ 6798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:50.457944 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:50.463880 (+ 5936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:50.464550 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:50.464680 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:50.470089 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:50.470097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:50.470102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:50.470151 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:50.473694 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:50.474824 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:50.474831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:50.474835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:50.474997 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:50.475619 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:50.475625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:50.475628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:50.476018 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:50.476149 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:50.477470 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0620 14:55:50.477753 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24445,"thread_start_us":24617,"threads_started":1}
I20260620 14:55:50.480882 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:50.415095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38403 (local address 127.21.159.193:59781)
0620 14:55:50.441179 (+ 26084us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:50.441259 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:50.447878 (+ 6619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:50.450678 (+ 2800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:50.450692 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:50.455464 (+ 4772us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:50.456498 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:50.456535 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:50.458257 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:50.458262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:50.463598 (+ 5336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:50.463628 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:50.465133 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:50.465144 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:50.465515 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:50.466448 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:50.466494 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:50.469709 (+ 3215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:50.473930 (+ 4221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:50.473942 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:50.473964 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:50.474660 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:50.475186 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:50.475191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:50.475195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:50.475471 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:50.476249 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:50.476259 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:50.476785 (+ 526us) client_negotiation.cc:769] Sending connection context
0620 14:55:50.478589 (+ 1804us) client_negotiation.cc:240] Negotiation successful
0620 14:55:50.479263 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25673,"thread_start_us":33104,"threads_started":1}
I20260620 14:55:50.483865 1242 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38403
I20260620 14:55:50.484264 1242 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:50.485255 1242 heartbeater.cc:511] Master 127.21.159.254:38403 requested a full tablet report, sending...
I20260620 14:55:50.488297 1029 ts_manager.cc:194] Registered new tserver with Master: b8160d6854034864af84586c00392952 (127.21.159.193:33969)
I20260620 14:55:50.491246 1029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:59781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
W20260620 14:55:50.510756 22143 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.21.159.194: real 0.110s user 0.000s sys 0.016s
I20260620 14:55:50.511942 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:58155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38403
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:55:50 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967349, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
W20260620 14:55:51.084146 1248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:51.084453 1248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:51.084625 1248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:51.092638 1248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:51.092775 1248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:51.092958 1248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:55:51.103803 1248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.159.212:58155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:55:51.105486 1248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:51.107126 1248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:51.122387 1262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:51.122515 1265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:51.122387 1263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:51.124442 1248 server_base.cc:1034] running on GCE node
I20260620 14:55:51.125240 1248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:51.126415 1248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:51.127681 1248 hybrid_clock.cc:648] HybridClock initialized: now 1781967351127619 us; error 54 us; skew 500 ppm
I20260620 14:55:51.128175 1248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:55:51 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967351, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:51.132586 1248 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:55:51.134702 1248 webserver.cc:460] Webserver started at http://127.21.159.194:38081/ using document root <none> and password file <none>
I20260620 14:55:51.135730 1248 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:51.135890 1248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:51.136243 1248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:55:51 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967351, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:55:51.156838 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "57fb2ed497c6490ca74de13a03349286"
format_stamp: "Formatted at 2026-06-20 14:55:51 on dist-test-slave-3mqx"
server_key: "33920a61802f179e23f74fdea6d4c290"
server_key_iv: "27495642dac394d8e6a9c6ead9626ada"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:51.158255 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "57fb2ed497c6490ca74de13a03349286"
format_stamp: "Formatted at 2026-06-20 14:55:51 on dist-test-slave-3mqx"
server_key: "33920a61802f179e23f74fdea6d4c290"
server_key_iv: "27495642dac394d8e6a9c6ead9626ada"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:51.167665 1248 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260620 14:55:51.187227 1272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:51.189175 1248 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 14:55:51.189412 1248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "57fb2ed497c6490ca74de13a03349286"
format_stamp: "Formatted at 2026-06-20 14:55:51 on dist-test-slave-3mqx"
server_key: "33920a61802f179e23f74fdea6d4c290"
server_key_iv: "27495642dac394d8e6a9c6ead9626ada"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:51.189687 1248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:51.210419 1248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:51.215214 1248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:51.215570 1248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:51.216784 1248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:51.218505 1248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:51.218621 1248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:51.218797 1248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:51.218879 1248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:51.280781 1248 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:35563
I20260620 14:55:51.280913 1384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:35563 every 8 connection(s)
I20260620 14:55:51.282431 1248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:55:51.291383 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 1248
I20260620 14:55:51.291575 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:55:51.292177 22143 external_mini_cluster.cc:1428] Setting key 19b8204baa053db409dd65f48cfee8ba
Jun 20 14:55:51 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967351, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:55:51.322876 1387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:51.286113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:52075 (local address 127.21.159.254:38403)
0620 14:55:51.301717 (+ 15604us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:51.301727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:51.301764 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:55:51.301843 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:51.301852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:51.301957 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:51.302140 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:51.304469 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.305548 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.307508 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.308347 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.308507 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:51.313320 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:51.313331 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:51.313340 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:51.313408 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:51.318418 (+ 5010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:51.319576 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:51.319587 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:51.319594 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:51.319785 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:51.320363 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:51.320373 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:51.320380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:51.320905 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:51.321118 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:51.322071 (+ 953us) server_negotiation.cc:299] Negotiation successful
0620 14:55:51.322380 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15405,"thread_start_us":16155,"threads_started":1}
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:55:51.323918 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:51.287251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38403 (local address 127.21.159.194:52075)
0620 14:55:51.290975 (+ 3724us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:51.291038 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:51.293756 (+ 2718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:51.302531 (+ 8775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:51.302546 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:51.303261 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:51.304184 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:51.304213 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.305775 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.305781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:51.307321 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:51.307345 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.308644 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.308652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:51.308986 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:51.309944 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:51.309975 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:51.312987 (+ 3012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:51.318646 (+ 5659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:51.318657 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:51.318678 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:51.319399 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:51.319932 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:51.319937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:51.319941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:51.320208 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:51.321079 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:51.321088 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:51.321557 (+ 469us) client_negotiation.cc:769] Sending connection context
0620 14:55:51.321878 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 14:55:51.322455 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3395,"thread_start_us":2938,"threads_started":1}
I20260620 14:55:51.325790 1385 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38403
I20260620 14:55:51.326159 1385 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:55:51.327137 1385 heartbeater.cc:511] Master 127.21.159.254:38403 requested a full tablet report, sending...
I20260620 14:55:51.329092 1029 ts_manager.cc:194] Registered new tserver with Master: 57fb2ed497c6490ca74de13a03349286 (127.21.159.194:35563)
I20260620 14:55:51.330238 1029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:52075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:55:51.358702 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:58155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38403
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:55:51.495814 1242 heartbeater.cc:503] Master 127.21.159.254:38403 was elected leader, sending a full tablet report...
W20260620 14:55:51.740520 1391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:51.740901 1391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:51.741086 1391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:51.748783 1391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:51.748975 1391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:51.749172 1391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:55:51.760310 1391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.159.212:58155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:38403
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:55:51.761890 1391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:51.763476 1391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:51.779100 1396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:51.779098 1397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:51.779204 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:51.780138 1391 server_base.cc:1034] running on GCE node
I20260620 14:55:51.780865 1391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:51.782037 1391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:51.783259 1391 hybrid_clock.cc:648] HybridClock initialized: now 1781967351783206 us; error 56 us; skew 500 ppm
I20260620 14:55:51.783635 1391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:55:51 dist-test-slave-3mqx krb5kdc[553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967351, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:51.790633 1391 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:55:51.792529 1391 webserver.cc:460] Webserver started at http://127.21.159.195:38127/ using document root <none> and password file <none>
I20260620 14:55:51.793318 1391 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:51.793483 1391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:51.793848 1391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:55:51 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967351, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:55:51.809859 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "8945ed3c48224c5d9a1851d922a46cab"
format_stamp: "Formatted at 2026-06-20 14:55:51 on dist-test-slave-3mqx"
server_key: "0f365326ad854a4c8b2459db067fe0d4"
server_key_iv: "9e52acc966be6ef00e85c3162f6532e4"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:51.810818 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "8945ed3c48224c5d9a1851d922a46cab"
format_stamp: "Formatted at 2026-06-20 14:55:51 on dist-test-slave-3mqx"
server_key: "0f365326ad854a4c8b2459db067fe0d4"
server_key_iv: "9e52acc966be6ef00e85c3162f6532e4"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:51.816936 1391 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 14:55:51.840476 1406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:51.842872 1391 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:55:51.843159 1391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "8945ed3c48224c5d9a1851d922a46cab"
format_stamp: "Formatted at 2026-06-20 14:55:51 on dist-test-slave-3mqx"
server_key: "0f365326ad854a4c8b2459db067fe0d4"
server_key_iv: "9e52acc966be6ef00e85c3162f6532e4"
server_key_version: "kuduclusterkey@0"
I20260620 14:55:51.843463 1391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:51.868381 1391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:51.874070 1391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:51.874496 1391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:51.875758 1391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:51.877650 1391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:51.877746 1391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:51.877872 1391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:51.877933 1391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:51.925850 1391 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:38739
I20260620 14:55:51.925865 1518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:38739 every 8 connection(s)
I20260620 14:55:51.927767 1391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:55:51.935895 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 1391
I20260620 14:55:51.936106 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:55:51.936672 22143 external_mini_cluster.cc:1428] Setting key 251c790c87af6066a10e73f12c55cafe
Jun 20 14:55:51 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967351, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:55:51.954476 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:51.930807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:37607 (local address 127.21.159.254:38403)
0620 14:55:51.931417 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:51.931424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:51.932673 (+ 1249us) server_negotiation.cc:407] Connection header received
0620 14:55:51.934695 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:51.934702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:51.934793 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:51.934929 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:51.937318 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.938527 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.940812 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.941504 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.941643 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:51.946535 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:51.946543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:51.946548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:51.946603 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:51.950186 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:51.951368 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:51.951374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:51.951378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:51.951575 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:51.952195 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:51.952201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:51.952204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:51.952628 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:51.952796 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:51.953782 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 14:55:51.954054 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":308,"threads_started":1}
I20260620 14:55:51.955641 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:51.931539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:38403 (local address 127.21.159.195:37607)
0620 14:55:51.932449 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:51.932507 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:51.934308 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:51.935236 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:51.935246 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:51.936048 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:51.937048 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:51.937079 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.938685 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.938693 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:51.940634 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:51.940649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.941746 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.941752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:51.942532 (+ 780us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:51.943522 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:51.943550 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:51.946114 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:51.950410 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:51.950446 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:51.950466 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:51.951195 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:51.951733 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:51.951738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:51.951742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:51.952041 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:51.952807 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:51.952819 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:51.953364 (+ 545us) client_negotiation.cc:769] Sending connection context
0620 14:55:51.953675 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 14:55:51.954309 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":393,"threads_started":1}
I20260620 14:55:51.957392 1519 heartbeater.cc:347] Connected to a master server at 127.21.159.254:38403
I20260620 14:55:51.957764 1519 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:51.958652 1519 heartbeater.cc:511] Master 127.21.159.254:38403 requested a full tablet report, sending...
I20260620 14:55:51.960482 1029 ts_manager.cc:194] Registered new tserver with Master: 8945ed3c48224c5d9a1851d922a46cab (127.21.159.195:38739)
I20260620 14:55:51.961673 1029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:37607
I20260620 14:55:51.966116 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:55:51.988101 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:51.971514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.21.159.254:38403)
0620 14:55:51.971696 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:51.971708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:51.972930 (+ 1222us) server_negotiation.cc:407] Connection header received
0620 14:55:51.973199 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:51.973206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:51.973302 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:51.973429 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:51.975091 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.975854 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.977338 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:51.977996 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:51.978140 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:51.981409 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:51.981438 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:51.981445 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:51.981500 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:51.984941 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:51.985792 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:51.985799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:51.985803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:51.985964 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:51.986413 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:51.986422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:51.986426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:51.986744 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:51.987032 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:51.987472 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:55:51.987736 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:55:51.993462 1029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:55:52.001888 1029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:55:52.035657 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.017475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33969 (local address 127.0.0.1:46502)
0620 14:55:52.018594 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:52.018643 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:52.018814 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:52.022124 (+ 3310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:52.022132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:52.022164 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:52.022615 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.022631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.025980 (+ 3349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.025986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.029657 (+ 3671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.029675 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.032486 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.032493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.032717 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.032787 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:55:52.032939 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:55:52.035288 (+ 2349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":637,"threads_started":1}
I20260620 14:55:52.038766 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.017520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.21.159.193:33969)
0620 14:55:52.019577 (+ 2057us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.019588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.019622 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:55:52.021268 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.021290 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.021609 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.021818 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:52.022854 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.025800 (+ 2946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.029874 (+ 4074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.031911 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.032119 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.032317 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.038225 (+ 5908us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.038438 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":1713,"threads_started":1}
I20260620 14:55:52.043359 1177 tablet_service.cc:1467] Processing CreateTablet for tablet 3645f2c23d4e4577ac4dd530c834f5c0 (DEFAULT_TABLE table=test-table [id=6d61331b427a41b0acf7d61dfd33e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:52.045485 1177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3645f2c23d4e4577ac4dd530c834f5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:52.051167 1536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.019845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:35563 (local address 127.0.0.1:56794)
0620 14:55:52.034914 (+ 15069us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:52.034943 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:52.035079 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:52.037568 (+ 2489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:52.037572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:52.037603 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:52.037991 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.038002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.042045 (+ 4043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.042050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.047456 (+ 5406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.047483 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.050064 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.050072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.050428 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.050466 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:55:52.050561 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:55:52.050716 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14930,"thread_start_us":29554,"threads_started":1}
I20260620 14:55:52.051357 1535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.020110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.21.159.194:35563)
0620 14:55:52.036654 (+ 16544us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.036664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.036694 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:55:52.036793 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.036801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.037107 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.037311 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:52.039033 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.041871 (+ 2838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.048099 (+ 6228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.049803 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.050008 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.050198 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.050770 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.050966 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15961,"thread_start_us":16277,"threads_started":1}
I20260620 14:55:52.051761 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.020608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:38739 (local address 127.0.0.1:49322)
0620 14:55:52.036340 (+ 15732us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:52.036359 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:52.036458 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:52.038300 (+ 1842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:52.038308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:52.038344 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:52.038630 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.038646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.041332 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.041341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.047721 (+ 6380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.047736 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.050872 (+ 3136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.050879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.051095 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.051131 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:55:52.051231 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:55:52.051391 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15627,"thread_start_us":13993,"threads_started":1}
I20260620 14:55:52.052628 1537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.020873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.21.159.195:38739)
0620 14:55:52.037208 (+ 16335us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.037221 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.037263 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:55:52.037382 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.037394 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.037715 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.037966 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:52.039107 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.041087 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.047962 (+ 6875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.050333 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.050518 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.050747 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.052109 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.052277 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16125,"thread_start_us":15971,"threads_started":1}
I20260620 14:55:52.056506 1320 tablet_service.cc:1467] Processing CreateTablet for tablet 3645f2c23d4e4577ac4dd530c834f5c0 (DEFAULT_TABLE table=test-table [id=6d61331b427a41b0acf7d61dfd33e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:52.058145 1320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3645f2c23d4e4577ac4dd530c834f5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:52.057932 1454 tablet_service.cc:1467] Processing CreateTablet for tablet 3645f2c23d4e4577ac4dd530c834f5c0 (DEFAULT_TABLE table=test-table [id=6d61331b427a41b0acf7d61dfd33e5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:52.059422 1454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3645f2c23d4e4577ac4dd530c834f5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:52.075145 1538 tablet_bootstrap.cc:492] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Bootstrap starting.
I20260620 14:55:52.080789 1538 tablet_bootstrap.cc:654] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:52.082160 1540 tablet_bootstrap.cc:492] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Bootstrap starting.
I20260620 14:55:52.082408 1538 log.cc:824] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:52.083930 1539 tablet_bootstrap.cc:492] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Bootstrap starting.
I20260620 14:55:52.086663 1540 tablet_bootstrap.cc:654] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:52.089133 1540 log.cc:824] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:52.089224 1539 tablet_bootstrap.cc:654] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:52.090816 1539 log.cc:824] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:52.094374 1538 tablet_bootstrap.cc:492] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: No bootstrap required, opened a new log
I20260620 14:55:52.094851 1538 ts_tablet_manager.cc:1397] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.011s
I20260620 14:55:52.097848 1539 tablet_bootstrap.cc:492] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: No bootstrap required, opened a new log
I20260620 14:55:52.098277 1539 ts_tablet_manager.cc:1397] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260620 14:55:52.109124 1539 raft_consensus.cc:348] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.110206 1539 raft_consensus.cc:374] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:52.110361 1539 raft_consensus.cc:729] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fb2ed497c6490ca74de13a03349286, State: Initialized, Role: FOLLOWER
I20260620 14:55:52.110334 1538 raft_consensus.cc:348] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.110949 1538 raft_consensus.cc:374] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:52.111093 1538 raft_consensus.cc:729] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8160d6854034864af84586c00392952, State: Initialized, Role: FOLLOWER
I20260620 14:55:52.111752 1538 consensus_queue.cc:260] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.113289 1540 tablet_bootstrap.cc:492] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: No bootstrap required, opened a new log
I20260620 14:55:52.113612 1540 ts_tablet_manager.cc:1397] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.020s
I20260620 14:55:52.113729 1538 ts_tablet_manager.cc:1428] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s
I20260620 14:55:52.114081 1539 consensus_queue.cc:260] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.122406 1539 ts_tablet_manager.cc:1428] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Time spent starting tablet: real 0.024s user 0.019s sys 0.000s
I20260620 14:55:52.122823 1385 heartbeater.cc:503] Master 127.21.159.254:38403 was elected leader, sending a full tablet report...
I20260620 14:55:52.126055 1540 raft_consensus.cc:348] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.126576 1540 raft_consensus.cc:374] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:52.126695 1540 raft_consensus.cc:729] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8945ed3c48224c5d9a1851d922a46cab, State: Initialized, Role: FOLLOWER
I20260620 14:55:52.127388 1540 consensus_queue.cc:260] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.128556 1519 heartbeater.cc:503] Master 127.21.159.254:38403 was elected leader, sending a full tablet report...
I20260620 14:55:52.129192 1540 ts_tablet_manager.cc:1428] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Time spent starting tablet: real 0.015s user 0.012s sys 0.005s
W20260620 14:55:52.179685 1520 tablet.cc:2369] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:55:52.202116 1244 tablet.cc:2369] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:55:52.292632 1386 tablet.cc:2369] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:55:52.514094 1546 raft_consensus.cc:482] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:55:52.514504 1546 raft_consensus.cc:504] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.516319 1546 leader_election.cc:288] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8160d6854034864af84586c00392952 (127.21.159.193:33969), 57fb2ed497c6490ca74de13a03349286 (127.21.159.194:35563)
I20260620 14:55:52.523986 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.516800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:49031 (local address 127.21.159.193:33969)
0620 14:55:52.517020 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.517027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.517658 (+ 631us) server_negotiation.cc:407] Connection header received
0620 14:55:52.517899 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.517905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.517991 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.518103 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:52.519163 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.520001 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.521960 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.522877 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.523052 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.523165 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.523522 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.523682 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:55:52.524113 1547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.516698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:33969 (local address 127.21.159.195:49031)
0620 14:55:52.517498 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:52.517514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:52.517745 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:52.518228 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:52.518235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:52.518321 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:52.518873 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.518889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.520147 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.520151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.521791 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.521804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.523040 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.523046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.523298 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.523346 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:55:52.523461 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:55:52.523625 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":4,"spinlock_wait_cycles":120576,"thread_start_us":494,"threads_started":1}
I20260620 14:55:52.525362 1535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.516800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:50243 (local address 127.21.159.194:35563)
0620 14:55:52.517020 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.517027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.517658 (+ 631us) server_negotiation.cc:407] Connection header received
0620 14:55:52.517900 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.517905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.517991 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.518103 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:52.519105 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.519880 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.522800 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.523928 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.524115 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.524200 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.524890 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.525069 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:55:52.525432 1548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.516673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:35563 (local address 127.21.159.195:50243)
0620 14:55:52.517481 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:52.517508 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:52.517720 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:52.518228 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:52.518235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:52.518278 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:52.518873 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.518889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.520044 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.520053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.522510 (+ 2457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:52.522530 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.524377 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.524384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:52.524638 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.524675 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:55:52.524834 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 14:55:52.524998 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":12,"thread_start_us":538,"threads_started":1}
I20260620 14:55:52.525496 1197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3645f2c23d4e4577ac4dd530c834f5c0" candidate_uuid: "8945ed3c48224c5d9a1851d922a46cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8160d6854034864af84586c00392952" is_pre_election: true
I20260620 14:55:52.526130 1197 raft_consensus.cc:2454] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8945ed3c48224c5d9a1851d922a46cab in term 0.
I20260620 14:55:52.526357 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3645f2c23d4e4577ac4dd530c834f5c0" candidate_uuid: "8945ed3c48224c5d9a1851d922a46cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fb2ed497c6490ca74de13a03349286" is_pre_election: true
I20260620 14:55:52.526830 1340 raft_consensus.cc:2454] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8945ed3c48224c5d9a1851d922a46cab in term 0.
I20260620 14:55:52.526964 1409 leader_election.cc:302] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8945ed3c48224c5d9a1851d922a46cab, b8160d6854034864af84586c00392952; no voters:
I20260620 14:55:52.527539 1546 raft_consensus.cc:2781] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:55:52.527707 1546 raft_consensus.cc:482] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:55:52.527823 1546 raft_consensus.cc:3037] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:52.530809 1546 raft_consensus.cc:504] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.531446 1546 leader_election.cc:288] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [CANDIDATE]: Term 1 election: Requested vote from peers b8160d6854034864af84586c00392952 (127.21.159.193:33969), 57fb2ed497c6490ca74de13a03349286 (127.21.159.194:35563)
I20260620 14:55:52.532071 1340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3645f2c23d4e4577ac4dd530c834f5c0" candidate_uuid: "8945ed3c48224c5d9a1851d922a46cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57fb2ed497c6490ca74de13a03349286"
I20260620 14:55:52.532095 1197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3645f2c23d4e4577ac4dd530c834f5c0" candidate_uuid: "8945ed3c48224c5d9a1851d922a46cab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8160d6854034864af84586c00392952"
I20260620 14:55:52.532299 1340 raft_consensus.cc:3037] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:52.532352 1197 raft_consensus.cc:3037] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:52.534615 1340 raft_consensus.cc:2454] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8945ed3c48224c5d9a1851d922a46cab in term 1.
I20260620 14:55:52.535010 1197 raft_consensus.cc:2454] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8945ed3c48224c5d9a1851d922a46cab in term 1.
I20260620 14:55:52.535250 1410 leader_election.cc:302] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57fb2ed497c6490ca74de13a03349286, 8945ed3c48224c5d9a1851d922a46cab; no voters:
I20260620 14:55:52.535629 1546 raft_consensus.cc:2781] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:52.536051 1546 raft_consensus.cc:686] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 1 LEADER]: Becoming Leader. State: Replica: 8945ed3c48224c5d9a1851d922a46cab, State: Running, Role: LEADER
I20260620 14:55:52.536577 1546 consensus_queue.cc:237] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } }
I20260620 14:55:52.541344 1029 catalog_manager.cc:5495] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab reported cstate change: term changed from 0 to 1, leader changed from <none> to 8945ed3c48224c5d9a1851d922a46cab (127.21.159.195). New cstate: current_term: 1 leader_uuid: "8945ed3c48224c5d9a1851d922a46cab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8945ed3c48224c5d9a1851d922a46cab" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 38739 } health_report { overall_health: HEALTHY } } }
I20260620 14:55:52.601001 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.594225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.21.159.195:38739)
0620 14:55:52.594834 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.594843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.595427 (+ 584us) server_negotiation.cc:407] Connection header received
0620 14:55:52.595657 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.595662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.595755 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.595918 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:52.596377 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.597180 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.598698 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.599198 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.599347 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.599882 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:52.600475 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:52.600588 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.600716 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.600785 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260620 14:55:52.610164 1197 raft_consensus.cc:1261] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 1 FOLLOWER]: Refusing update from remote peer 8945ed3c48224c5d9a1851d922a46cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:52.610168 1340 raft_consensus.cc:1261] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 1 FOLLOWER]: Refusing update from remote peer 8945ed3c48224c5d9a1851d922a46cab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:52.611303 1549 consensus_queue.cc:1035] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8160d6854034864af84586c00392952" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 33969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:52.611645 1546 consensus_queue.cc:1035] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [LEADER]: Connected to new peer: Peer: permanent_uuid: "57fb2ed497c6490ca74de13a03349286" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 35563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:52.623746 1557 mvcc.cc:205] Tried to move back new op lower bound from 7298938276280889344 to 7298938275994398720. Current Snapshot: MvccSnapshot[applied={T|T < 7298938276280889344}]
I20260620 14:55:52.637833 1029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:52.638135 1029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:52.641901 1029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b412093d15ea4fd0a0ab86c007f4d42f: Sending DeleteTablet for 3 replicas of tablet 3645f2c23d4e4577ac4dd530c834f5c0
I20260620 14:55:52.643229 1454 tablet_service.cc:1514] Processing DeleteTablet for tablet 3645f2c23d4e4577ac4dd530c834f5c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:52 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:49322
I20260620 14:55:52.643210 1320 tablet_service.cc:1514] Processing DeleteTablet for tablet 3645f2c23d4e4577ac4dd530c834f5c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:52 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56794
I20260620 14:55:52.644061 1177 tablet_service.cc:1514] Processing DeleteTablet for tablet 3645f2c23d4e4577ac4dd530c834f5c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:52 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:46502
I20260620 14:55:52.644812 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 1109
I20260620 14:55:52.655419 1567 tablet_replica.cc:330] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: stopping tablet replica
I20260620 14:55:52.655673 1569 tablet_replica.cc:330] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: stopping tablet replica
I20260620 14:55:52.656296 1569 raft_consensus.cc:2229] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:55:52.656916 1569 raft_consensus.cc:2258] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:52.656917 1567 raft_consensus.cc:2229] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:55:52.657869 1568 tablet_replica.cc:330] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: stopping tablet replica
I20260620 14:55:52.658367 1568 raft_consensus.cc:2229] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:55:52.658375 1567 raft_consensus.cc:2258] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:52.658898 1568 raft_consensus.cc:2258] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:52.661190 1568 ts_tablet_manager.cc:1905] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:52.661904 1567 ts_tablet_manager.cc:1905] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:52.662791 1569 ts_tablet_manager.cc:1905] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:52.669634 1568 ts_tablet_manager.cc:1918] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:52.669884 1568 log.cc:1196] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/3645f2c23d4e4577ac4dd530c834f5c0
I20260620 14:55:52.670260 1567 ts_tablet_manager.cc:1918] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:52.670485 1567 log.cc:1196] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/3645f2c23d4e4577ac4dd530c834f5c0
I20260620 14:55:52.670497 1568 ts_tablet_manager.cc:1939] T 3645f2c23d4e4577ac4dd530c834f5c0 P 57fb2ed497c6490ca74de13a03349286: Deleting consensus metadata
I20260620 14:55:52.671109 1567 ts_tablet_manager.cc:1939] T 3645f2c23d4e4577ac4dd530c834f5c0 P 8945ed3c48224c5d9a1851d922a46cab: Deleting consensus metadata
Jun 20 14:55:52 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:55:52.672706 1016 catalog_manager.cc:4841] TS 57fb2ed497c6490ca74de13a03349286 (127.21.159.194:35563): tablet 3645f2c23d4e4577ac4dd530c834f5c0 (table test-table [id=6d61331b427a41b0acf7d61dfd33e5c3]) successfully deleted
I20260620 14:55:52.672797 1015 catalog_manager.cc:4841] TS 8945ed3c48224c5d9a1851d922a46cab (127.21.159.195:38739): tablet 3645f2c23d4e4577ac4dd530c834f5c0 (table test-table [id=6d61331b427a41b0acf7d61dfd33e5c3]) successfully deleted
I20260620 14:55:52.677994 1569 ts_tablet_manager.cc:1918] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:52.678303 1569 log.cc:1196] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/3645f2c23d4e4577ac4dd530c834f5c0
I20260620 14:55:52.678913 1569 ts_tablet_manager.cc:1939] T 3645f2c23d4e4577ac4dd530c834f5c0 P b8160d6854034864af84586c00392952: Deleting consensus metadata
I20260620 14:55:52.680790 1015 catalog_manager.cc:4841] TS b8160d6854034864af84586c00392952 (127.21.159.193:33969): tablet 3645f2c23d4e4577ac4dd530c834f5c0 (table test-table [id=6d61331b427a41b0acf7d61dfd33e5c3]) successfully deleted
I20260620 14:55:52.682981 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.645837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.21.159.193:33969)
0620 14:55:52.646020 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.646026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.652298 (+ 6272us) server_negotiation.cc:407] Connection header received
0620 14:55:52.652564 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.652572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.652670 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.652789 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:52.656786 (+ 3997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.658050 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.663071 (+ 5021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.663752 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.663868 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.674262 (+ 10394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:52.674285 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:52.674304 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:52.674362 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:52.679466 (+ 5104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:52.680581 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:52.680591 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:52.680595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:52.680753 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:52.681264 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:52.681268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:52.681270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:52.681806 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:52.682062 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.682476 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.682651 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:55:52.683809 1237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:52.776501 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 1109
I20260620 14:55:52.803013 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 1248
Jun 20 14:55:52 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:55:52.821856 1535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.804025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.21.159.194:35563)
0620 14:55:52.804199 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.804203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.804283 (+ 80us) server_negotiation.cc:407] Connection header received
0620 14:55:52.804480 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.804484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.804546 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.804631 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:52.806449 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.807242 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.809388 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.809798 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.809910 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.815707 (+ 5797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:52.815723 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:52.815739 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:52.815784 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:52.818758 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:52.819662 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:52.819676 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:52.819679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:52.819817 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:52.820230 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:52.820235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:52.820238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:52.820623 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:52.820923 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.821268 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.821490 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:55:52.822570 1380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:52.914588 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 1248
I20260620 14:55:52.939246 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 1391
Jun 20 14:55:52 dist-test-slave-3mqx krb5kdc[553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:55:52.960486 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:52.940142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.21.159.195:38739)
0620 14:55:52.940338 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:52.940345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:52.940558 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:55:52.940759 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:52.940765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:52.940857 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:52.940974 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:52.942910 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.944426 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.946683 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:52.947546 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:52.947696 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:52.953872 (+ 6176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:52.953887 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:52.953903 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:52.953943 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:52.957294 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:52.958130 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:52.958146 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:52.958148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:52.958308 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:52.958731 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:52.958736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:52.958741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:52.959209 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:52.959514 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:52.959871 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:55:52.960114 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:55:52.961159 1514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:53.087904 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 1391
I20260620 14:55:53.107934 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 997
I20260620 14:55:53.109323 1059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:55:53.224325 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 997
I20260620 14:55:53.249464 22143 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:55:53.585673 22143 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:55:53.585976 22143 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:55:53.932603 22143 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:55:53.933 UTC [583] LOG: received smart shutdown request
2026-06-20 14:55:53.936 UTC [583] LOG: background worker "logical replication launcher" (PID 592) exited with exit code 1
2026-06-20 14:55:53.937 UTC [587] LOG: shutting down
2026-06-20 14:55:54.117 UTC [583] LOG: database system is shut down
2026-06-20T14:55:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53448 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:54 dist-test-slave-3mqx krb5kdc[1583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:54 dist-test-slave-3mqx krb5kdc[1583](info): set up 2 sockets
Jun 20 14:55:54 dist-test-slave-3mqx krb5kdc[1583](info): commencing operation
krb5kdc: starting...
W20260620 14:55:55.439958 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:55:55 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/rangeradmin_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/rangeradmin_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/rangerlookup_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/rangerlookup_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
I20260620 14:55:55.786993 22143 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskmvfD4B/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/postgres -l logfile start
2026-06-20 14:55:58.510 UTC [1613] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:55:58.510 UTC [1613] LOG: listening on IPv4 address "127.21.159.212", port 52119
2026-06-20 14:55:58.514 UTC [1613] LOG: listening on Unix socket "/tmp/.s.PGSQL.52119"
2026-06-20 14:55:58.540 UTC [1616] LOG: database system was shut down at 2026-06-20 14:55:56 UTC
2026-06-20 14:55:58.544 UTC [1613] LOG: database system is ready to accept connections
I20260620 14:55:59.530901 22143 mini_postgres.cc:96] Postgres bound to 52119
2026-06-20 14:55:59.548 UTC [1624] FATAL: database "slave" does not exist
127.21.159.212:52119 - accepting connections
I20260620 14:55:59.548748 22143 mini_ranger.cc:161] Starting Ranger...
I20260620 14:55:59.576472 22143 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:56:00.116111 22143 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:56:00.116262 22143 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-admin
2026-06-20 14:56:00,412 [I] DB FLAVOR :POSTGRES
2026-06-20 14:56:00,412 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:56:00,413 [I] Checking connection..
2026-06-20 14:56:00,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:56:00,696 [I] Checking connection passed.
2026-06-20 14:56:00,696 [I] --------- Verifying version history table ---------
2026-06-20 14:56:00,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:56:00,959 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:56:00,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:56:01,243 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:56:01,244 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:56:01,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:56:01.506 UTC [1688] WARNING: there is no transaction in progress
2026-06-20 14:56:01,513 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:56:01,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:56:01,810 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:56:01,810 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:56:01,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:56:02,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:56:02,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3mqx.c.gcp-upstream.internal','N') ;"
2026-06-20 14:56:02,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:56:02,857 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:56:02,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:56:03,101 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:56:03,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:56:03,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:56:03,602 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:56:03,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:56:04.642 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.658 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.675 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.691 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.711 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.733 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.804 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.813 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.829 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.841 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.852 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.866 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.876 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.883 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.891 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:04.898 UTC [1796] WARNING: there is no transaction in progress
2026-06-20 14:56:05,612 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:56:05,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:56:05,862 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:56:05,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:56:06,145 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:56:06,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:56:06,406 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:56:06,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:56:06,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:56:06,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3mqx.c.gcp-upstream.internal';"
2026-06-20 14:56:07,154 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:56:07,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:56:07,421 [I] DB_PATCHES have already been applied
I20260620 14:56:07.428298 22143 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-admin:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskmvfD4B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:56:07.428390 22143 mini_ranger.cc:193] Using host: 127.21.159.212
I20260620 14:56:07.442598 22143 mini_ranger.cc:227] Ranger admin URL: http://127.21.159.212:43855
Jun 20, 2026 2:56:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:56:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:56:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:56:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:56:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:56:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:56:08.700928 22143 mini_ranger.cc:160] Time spent starting Ranger: real 9.152s user 0.000s sys 0.041s
Jun 20, 2026 2:56:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:56:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:56:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:56:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:56:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:56:30.404561 22143 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.159.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.159.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/rangerkms_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.159.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/rangerkms_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.159.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.159.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.159.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/HTTP_127.21.159.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:56:30 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967390, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:56:30.581166 22143 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:56:30.603789 22143 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:56:31.527248 22143 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:56:31.527341 22143 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms
I20260620 14:56:31.527371 22143 mini_ranger_kms.cc:227] Using postgres at 127.21.159.212:52119
2026-06-20 14:56:31,899 [I] DB FLAVOR :POSTGRES
2026-06-20 14:56:31,899 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:56:31,900 [I] Checking connection
2026-06-20 14:56:31,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:56:32,158 [I] connection success
2026-06-20 14:56:32,158 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:56:32,158 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:56:32,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:56:32,400 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:56:32,401 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:56:32,401 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:56:32,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.159.212:52119/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:56:32,661 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:56:33.152987 22143 mini_ranger_kms.cc:310] Created kms service
I20260620 14:56:33.352105 22143 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:56:33.400259 22143 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:56:33 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967390, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:33.872681 22143 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:56:33.873090 22143 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms:/tmp/dist-test-taskmvfD4B/build/asan/bin/postgresql.jar:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskmvfD4B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:56:33.873163 22143 mini_ranger_kms.cc:240] Using host: 127.21.159.212
I20260620 14:56:33.891340 22143 mini_ranger_kms.cc:276] Ranger KMS PID: 1976
I20260620 14:56:33.891486 22143 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.159.212:33583
Jun 20, 2026 2:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:56:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:56:36.314241 22143 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.733s user 0.104s sys 0.280s
I20260620 14:56:36.314510 22143 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:56:36.314680 22143 mini_ranger_kms.cc:401] 127.21.159.212:33583/kms/v1/keys
Jun 20 14:56:36 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967396, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.159.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:56:36 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967396, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.159.212@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
Jun 20 14:56:37 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967396, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.159.212@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
Jun 20 14:56:39 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:56:39.671365 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35385
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:35385
--ranger_config_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:40.026661 2027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:40.026968 2027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:40.027177 2027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:40.034629 2027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:56:40.034752 2027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:40.034832 2027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:40.034919 2027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:56:40.035001 2027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:56:40.045615 2027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.159.212:33583/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:35385
--ranger_config_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:40.047099 2027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:40.048576 2027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:40.061230 2033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:40.061203 2035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:40.061291 2032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:40.061882 2027 server_base.cc:1034] running on GCE node
I20260620 14:56:40.062714 2027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:40.064224 2027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:40.065431 2027 hybrid_clock.cc:648] HybridClock initialized: now 1781967400065401 us; error 42 us; skew 500 ppm
I20260620 14:56:40.065761 2027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:40 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967400, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:40.069249 2027 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:56:40.070842 2027 webserver.cc:460] Webserver started at http://127.21.159.254:33455/ using document root <none> and password file <none>
I20260620 14:56:40.071576 2027 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:40.071713 2027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:40.072042 2027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:56:40 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967400, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:40.139631 2027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "100ddaa2de30429193b9345a76d61fcb"
format_stamp: "Formatted at 2026-06-20 14:56:40 on dist-test-slave-3mqx"
server_key: "da48cb37d7fcc2ae8ef73a9db6ed634c"
server_key_iv: "fdbca1386ad26a6963fa62fbe0f5b04f"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:40.140642 2027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "100ddaa2de30429193b9345a76d61fcb"
format_stamp: "Formatted at 2026-06-20 14:56:40 on dist-test-slave-3mqx"
server_key: "da48cb37d7fcc2ae8ef73a9db6ed634c"
server_key_iv: "fdbca1386ad26a6963fa62fbe0f5b04f"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:40.147003 2027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 14:56:40.168596 2042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:40.170310 2027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:56:40.170502 2027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "100ddaa2de30429193b9345a76d61fcb"
format_stamp: "Formatted at 2026-06-20 14:56:40 on dist-test-slave-3mqx"
server_key: "da48cb37d7fcc2ae8ef73a9db6ed634c"
server_key_iv: "fdbca1386ad26a6963fa62fbe0f5b04f"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:40.170750 2027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:40.183790 2027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:40.188047 2027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:40.188400 2027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:40.209758 2027 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:35385
I20260620 14:56:40.209755 2093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:35385 every 8 connection(s)
I20260620 14:56:40.211359 2027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:56:40.215302 2094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:40.219551 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2027
I20260620 14:56:40.219794 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:56:40.220546 22143 external_mini_cluster.cc:1428] Setting key f062e11dfdd6e884a4dd10b79cc74966
I20260620 14:56:40.226747 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Bootstrap starting.
I20260620 14:56:40.229976 2094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:40.231099 2094 log.cc:824] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:40.234196 2094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: No bootstrap required, opened a new log
Jun 20 14:56:40 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:40.243906 2094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:40.244210 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:40.222702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.21.159.254:35385)
0620 14:56:40.223600 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:40.223613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:40.223661 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:56:40.225460 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:40.225486 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:40.225907 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:40.226321 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:40.227787 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:40.229124 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:40.230485 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:40.231255 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:40.231436 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:40.235878 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:40.235889 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:40.235906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:40.235952 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:40.239512 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:40.240413 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:40.240421 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:40.240428 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:40.240590 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:40.241003 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:40.241007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:40.241010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:40.241679 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:40.241934 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:40.242511 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 14:56:40.242986 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":445,"threads_started":1}
I20260620 14:56:40.244307 2094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:40.244403 2094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100ddaa2de30429193b9345a76d61fcb, State: Initialized, Role: FOLLOWER
I20260620 14:56:40.245138 2094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:40.245375 2094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:56:40.245505 2094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:56:40.245690 2094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:40.248020 2094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:40.248514 2094 leader_election.cc:302] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100ddaa2de30429193b9345a76d61fcb; no voters:
I20260620 14:56:40.249155 2094 leader_election.cc:288] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:56:40.249353 2099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:56:40.250116 2099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 LEADER]: Becoming Leader. State: Replica: 100ddaa2de30429193b9345a76d61fcb, State: Running, Role: LEADER
I20260620 14:56:40.250689 2099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:40.251302 2094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:56:40.253765 2100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100ddaa2de30429193b9345a76d61fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } } }
I20260620 14:56:40.254174 2100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:40.255743 2101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100ddaa2de30429193b9345a76d61fcb. Latest consensus state: current_term: 1 leader_uuid: "100ddaa2de30429193b9345a76d61fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } } }
I20260620 14:56:40.256105 2101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:40.258378 2105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:56:40.265141 2105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:56:40.265952 2094 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:56:40.287577 2105 catalog_manager.cc:1347] Generated new cluster ID: 1edc513b5ef149c7aeda570358c8d449
I20260620 14:56:40.287762 2105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:56:40.320684 2105 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:56:40.322749 2105 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:56:40.333190 2105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Generated new TSK 0
I20260620 14:56:40.334359 2105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:56:41.084852 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:35385
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:56:41 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:56:41.546818 2139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:41.547284 2139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:41.547451 2139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:41.559738 2139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:41.559916 2139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:41.560153 2139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:56:41.578295 2139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.159.212:33583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:41.580225 2139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:41.582130 2139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:41.606433 2150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:41.612020 2149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:41.612473 2139 server_base.cc:1034] running on GCE node
W20260620 14:56:41.613458 2152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:41.614450 2139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:41.615963 2139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:41.617223 2139 hybrid_clock.cc:648] HybridClock initialized: now 1781967401617144 us; error 68 us; skew 500 ppm
I20260620 14:56:41.617646 2139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:41 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:41.622396 2139 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:56:41.625202 2139 webserver.cc:460] Webserver started at http://127.21.159.193:38619/ using document root <none> and password file <none>
I20260620 14:56:41.626276 2139 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:41.626444 2139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:41.626888 2139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:56:41 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:41.648102 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "188a1fb6a1c64d43aca3a596b111f4d4"
format_stamp: "Formatted at 2026-06-20 14:56:41 on dist-test-slave-3mqx"
server_key: "4a2adfd50ae2cd1a0512b1b8e66c3ac2"
server_key_iv: "d26b035cfbf704514a693834e1270f10"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:41.649247 2139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "188a1fb6a1c64d43aca3a596b111f4d4"
format_stamp: "Formatted at 2026-06-20 14:56:41 on dist-test-slave-3mqx"
server_key: "4a2adfd50ae2cd1a0512b1b8e66c3ac2"
server_key_iv: "d26b035cfbf704514a693834e1270f10"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:41.656636 2139 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 14:56:41.691819 2159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:41.695561 2139 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.011s sys 0.004s
I20260620 14:56:41.695797 2139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "188a1fb6a1c64d43aca3a596b111f4d4"
format_stamp: "Formatted at 2026-06-20 14:56:41 on dist-test-slave-3mqx"
server_key: "4a2adfd50ae2cd1a0512b1b8e66c3ac2"
server_key_iv: "d26b035cfbf704514a693834e1270f10"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:41.696074 2139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:41.729808 2139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:41.735602 2139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:41.735970 2139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:41.740594 2139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:41.742772 2139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:41.742882 2139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:41.743026 2139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:41.743115 2139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:56:41 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:41.938192 2139 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:43989
I20260620 14:56:41.938266 2271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:43989 every 8 connection(s)
I20260620 14:56:41.939949 2139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:56:41.947507 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2139
I20260620 14:56:41.947809 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:56:41.948814 22143 external_mini_cluster.cc:1428] Setting key 6000f5ff20c8e7302f389b92cc4610e8
Jun 20 14:56:41 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:41.986056 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:41.943168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:55631 (local address 127.21.159.254:35385)
0620 14:56:41.943867 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:41.943872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:41.967836 (+ 23964us) server_negotiation.cc:407] Connection header received
0620 14:56:41.970507 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:41.970512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:41.970589 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:41.970736 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:41.972614 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:41.973458 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:41.975242 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:41.975688 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:41.975828 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:41.979351 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:41.979356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:41.979360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:41.979394 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:41.982170 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:41.983071 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:41.983077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:41.983081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:41.983218 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:41.983800 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:41.983805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:41.983808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:41.984174 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:41.984307 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:41.985396 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0620 14:56:41.985652 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":454,"threads_started":1}
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:56:41.988521 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:41.944571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:35385 (local address 127.21.159.193:55631)
0620 14:56:41.967626 (+ 23055us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:41.967691 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:41.970132 (+ 2441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:41.971031 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:41.971041 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:41.971572 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:41.972235 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:41.972255 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:41.973629 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:41.973634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:41.975020 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:41.975045 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:41.975938 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:41.975945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:41.976192 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:41.976921 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:41.976945 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:41.979044 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:41.982338 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:41.982348 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:41.982370 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:41.982914 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:41.983350 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:41.983355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:41.983358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:41.983667 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:41.984419 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:41.984428 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:41.984914 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 14:56:41.986368 (+ 1454us) client_negotiation.cc:240] Negotiation successful
0620 14:56:41.986963 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22695,"thread_start_us":22474,"threads_started":1}
I20260620 14:56:41.990504 2272 heartbeater.cc:347] Connected to a master server at 127.21.159.254:35385
I20260620 14:56:41.990866 2272 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:41.991802 2272 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
I20260620 14:56:41.994448 2059 ts_manager.cc:194] Registered new tserver with Master: 188a1fb6a1c64d43aca3a596b111f4d4 (127.21.159.193:43989)
I20260620 14:56:41.996781 2059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:55631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:56:42.028954 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:35385
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:56:42 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967401, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
W20260620 14:56:42.507826 2278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:42.508270 2278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:42.508416 2278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:42.516943 2278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:42.517076 2278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:42.517361 2278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:56:42.528476 2278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.159.212:33583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:42.529896 2278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:42.531322 2278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:42.545836 2292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:42.545877 2293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:42.546542 2295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:42.547017 2278 server_base.cc:1034] running on GCE node
I20260620 14:56:42.547787 2278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:42.548856 2278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:42.550082 2278 hybrid_clock.cc:648] HybridClock initialized: now 1781967402550021 us; error 60 us; skew 500 ppm
I20260620 14:56:42.550411 2278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:42 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967402, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:42.557127 2278 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:56:42.558806 2278 webserver.cc:460] Webserver started at http://127.21.159.194:43419/ using document root <none> and password file <none>
I20260620 14:56:42.559613 2278 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:42.559780 2278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:42.560212 2278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:56:42 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967402, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:42.577792 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "873acfa943bf4544854a0a2bb4fd517d"
format_stamp: "Formatted at 2026-06-20 14:56:42 on dist-test-slave-3mqx"
server_key: "67853cb0bb137ee40c09ae396a3d2ea5"
server_key_iv: "c5e005ecd25e0b2731e80c9a5894e5db"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:42.578866 2278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "873acfa943bf4544854a0a2bb4fd517d"
format_stamp: "Formatted at 2026-06-20 14:56:42 on dist-test-slave-3mqx"
server_key: "67853cb0bb137ee40c09ae396a3d2ea5"
server_key_iv: "c5e005ecd25e0b2731e80c9a5894e5db"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:42.586179 2278 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 14:56:42.618717 2302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:42.621083 2278 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.005s
I20260620 14:56:42.621325 2278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "873acfa943bf4544854a0a2bb4fd517d"
format_stamp: "Formatted at 2026-06-20 14:56:42 on dist-test-slave-3mqx"
server_key: "67853cb0bb137ee40c09ae396a3d2ea5"
server_key_iv: "c5e005ecd25e0b2731e80c9a5894e5db"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:42.621635 2278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:42.669050 2278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:42.674727 2278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:42.675103 2278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:42.676383 2278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:42.678373 2278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:42.678486 2278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:42.678642 2278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:42.678719 2278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:42.725793 2278 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:46429
I20260620 14:56:42.725852 2414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:46429 every 8 connection(s)
I20260620 14:56:42.727690 2278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:56:42.733996 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2278
I20260620 14:56:42.734189 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:56:42.734800 22143 external_mini_cluster.cc:1428] Setting key 4daf169a913954ce262384134017048f
Jun 20 14:56:42 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967402, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:56:42.774577 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:42.730872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:36407 (local address 127.21.159.254:35385)
0620 14:56:42.749513 (+ 18641us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:42.749522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:42.754625 (+ 5103us) server_negotiation.cc:407] Connection header received
0620 14:56:42.754705 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:42.754710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:42.754813 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:42.754928 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:42.757381 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:42.758448 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:42.760909 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:42.761558 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:42.761726 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:42.766079 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:42.766086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:42.766091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:42.766134 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:42.769663 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:42.770682 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:42.770687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:42.770692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:42.770830 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:42.771375 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:42.771380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:42.771383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:42.771760 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:42.771896 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:42.773871 (+ 1975us) server_negotiation.cc:299] Negotiation successful
0620 14:56:42.774144 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18466,"thread_start_us":18631,"threads_started":1}
I20260620 14:56:42.774683 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:42.731568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:35385 (local address 127.21.159.194:36407)
0620 14:56:42.751803 (+ 20235us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:42.751865 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:42.754193 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:42.755342 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:42.755358 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:42.756080 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:42.757118 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:42.757147 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:42.758654 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:42.758659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:42.760724 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:42.760739 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:42.761830 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:42.761837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:42.762130 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:42.762985 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:42.763015 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:42.765777 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:42.769852 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:42.769862 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:42.769883 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:42.770479 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:42.770949 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:42.770954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:42.770957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:42.771246 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:42.771964 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:42.771973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:42.772408 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 14:56:42.772708 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 14:56:42.773235 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19887,"thread_start_us":19624,"threads_started":1}
I20260620 14:56:42.776542 2415 heartbeater.cc:347] Connected to a master server at 127.21.159.254:35385
I20260620 14:56:42.776937 2415 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:42.777853 2415 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
I20260620 14:56:42.779670 2059 ts_manager.cc:194] Registered new tserver with Master: 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429)
I20260620 14:56:42.780764 2059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:36407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:56:42.813987 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:35385
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:56:43.001362 2272 heartbeater.cc:503] Master 127.21.159.254:35385 was elected leader, sending a full tablet report...
W20260620 14:56:43.179389 2421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:43.179682 2421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:43.179891 2421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:43.188304 2421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:43.188439 2421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:43.188642 2421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:56:43.200104 2421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.159.212:33583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:43.201732 2421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:43.203365 2421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:43.218029 2426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:43.218066 2427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:43.219381 2429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:43.221299 2421 server_base.cc:1034] running on GCE node
I20260620 14:56:43.221967 2421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:43.223028 2421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:43.224325 2421 hybrid_clock.cc:648] HybridClock initialized: now 1781967403224251 us; error 77 us; skew 500 ppm
I20260620 14:56:43.224668 2421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:43 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967403, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:43.228371 2421 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:56:43.229962 2421 webserver.cc:460] Webserver started at http://127.21.159.195:43619/ using document root <none> and password file <none>
I20260620 14:56:43.230849 2421 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:43.230993 2421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:43.231331 2421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:56:43 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967403, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:43.250864 2421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad"
format_stamp: "Formatted at 2026-06-20 14:56:43 on dist-test-slave-3mqx"
server_key: "2b6f5cf801364e6c45ec8b5fc9ec5f1f"
server_key_iv: "b3f214b3558cba6ae1ba5bad3afa1d9c"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:43.251879 2421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad"
format_stamp: "Formatted at 2026-06-20 14:56:43 on dist-test-slave-3mqx"
server_key: "2b6f5cf801364e6c45ec8b5fc9ec5f1f"
server_key_iv: "b3f214b3558cba6ae1ba5bad3afa1d9c"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:43.257495 2421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:56:43.274540 2436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:43.276566 2421 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:56:43.276768 2421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad"
format_stamp: "Formatted at 2026-06-20 14:56:43 on dist-test-slave-3mqx"
server_key: "2b6f5cf801364e6c45ec8b5fc9ec5f1f"
server_key_iv: "b3f214b3558cba6ae1ba5bad3afa1d9c"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:43.277141 2421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:43.306479 2421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:43.311223 2421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:43.311714 2421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:43.312915 2421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:43.314536 2421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:43.314620 2421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:43.314718 2421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:43.314801 2421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:43.350101 2421 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:34903
I20260620 14:56:43.350145 2548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:34903 every 8 connection(s)
I20260620 14:56:43.351624 2421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:56:43.354027 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2421
I20260620 14:56:43.354233 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:56:43.354912 22143 external_mini_cluster.cc:1428] Setting key 014576d22b1c64466fc6a175e3c67535
Jun 20 14:56:43 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967403, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:43.372854 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:43.354595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:60417 (local address 127.21.159.254:35385)
0620 14:56:43.355214 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:43.355220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:43.356429 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 14:56:43.358816 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:43.358821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:43.358916 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:43.359034 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:43.360654 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.361456 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.363155 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.363577 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.363709 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:43.367009 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:43.367014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:43.367017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:43.367096 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:43.369762 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.370513 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.370519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.370523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.370680 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.371138 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.371145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.371153 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.371409 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:43.371575 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:43.372266 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 14:56:43.372492 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":289,"threads_started":1}
I20260620 14:56:43.373849 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:43.355334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:35385 (local address 127.21.159.195:60417)
0620 14:56:43.356242 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:43.356299 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:43.358495 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:43.359305 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:43.359314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:43.359784 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:43.360481 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:43.360500 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.361636 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.361640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:43.362977 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:43.362988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.363822 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.363829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:43.364057 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:43.364716 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:43.364742 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:43.366770 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:43.369909 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:43.369919 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:43.369939 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:43.370380 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:43.370796 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:43.370801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:43.370805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:43.370995 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:43.371550 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:43.371558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:43.371909 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 14:56:43.372183 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 14:56:43.372719 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":397,"threads_started":1}
I20260620 14:56:43.375388 2549 heartbeater.cc:347] Connected to a master server at 127.21.159.254:35385
I20260620 14:56:43.375663 2549 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:43.376299 2549 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
I20260620 14:56:43.377781 2059 ts_manager.cc:194] Registered new tserver with Master: 79108d4bdcfd4aa6b6c2f0d92bc3c3ad (127.21.159.195:34903)
I20260620 14:56:43.378533 2059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:60417
I20260620 14:56:43.383932 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:56:43.385710 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2139
Jun 20 14:56:43 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:56:43.403100 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:43.386596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.21.159.193:43989)
0620 14:56:43.387216 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:43.387225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:43.387256 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:56:43.387389 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:43.387397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:43.387635 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:43.387806 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:43.389427 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.390423 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.391994 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.392551 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.392689 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:43.397108 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:43.397120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:43.397134 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:43.397168 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:43.400078 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.400802 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.400809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.400812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.400970 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.401413 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.401419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.401425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.401770 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:43.402101 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:43.402398 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 14:56:43.402665 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":252,"threads_started":1}
I20260620 14:56:43.404315 2267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:43.487895 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2139
I20260620 14:56:43.503890 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2278
Jun 20 14:56:43 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:56:43.520314 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:43.504819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.21.159.194:46429)
0620 14:56:43.505357 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:43.505364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:43.505390 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:56:43.505463 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:43.505470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:43.505673 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:43.505806 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:43.507103 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.508132 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.509434 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.509991 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.510087 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:43.514332 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:43.514344 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:43.514358 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:43.514397 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:43.517344 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.518074 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.518082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.518091 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.518261 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.518640 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.518645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.518647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.519236 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:43.519491 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:43.519751 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:56:43.519989 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":285,"threads_started":1}
I20260620 14:56:43.521610 2410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:43.605878 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2278
I20260620 14:56:43.625937 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2421
Jun 20 14:56:43 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:56:43.643953 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:43.627202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.21.159.195:34903)
0620 14:56:43.627878 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:43.627885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:43.627915 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:56:43.627981 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:43.627996 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:43.628203 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:43.628332 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:43.629766 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.630864 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.632528 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:43.633112 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:43.633213 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:43.638065 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:43.638078 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:43.638094 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:43.638132 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:43.640983 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.641827 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.641835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.641837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.641968 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:43.642377 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:43.642380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:43.642382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:43.642727 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:43.642969 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:43.643385 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 14:56:43.643590 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":371,"threads_started":1}
I20260620 14:56:43.645519 2544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:43.729688 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2421
I20260620 14:56:43.746168 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2027
I20260620 14:56:43.747700 2089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:43.854076 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2027
I20260620 14:56:43.878420 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35385
--webserver_interface=127.21.159.254
--webserver_port=33455
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:35385
--ranger_config_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:44.202811 2561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:44.203142 2561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:44.203300 2561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:44.210510 2561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:56:44.210620 2561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:44.210700 2561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:44.210812 2561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:56:44.210881 2561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:56:44.221172 2561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:35385
--ranger_config_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=33455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:44.222524 2561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:44.223933 2561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:44.236380 2567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:44.236475 2566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:44.236430 2569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:44.237625 2561 server_base.cc:1034] running on GCE node
I20260620 14:56:44.238463 2561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:44.240186 2561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:44.241391 2561 hybrid_clock.cc:648] HybridClock initialized: now 1781967404241342 us; error 53 us; skew 500 ppm
I20260620 14:56:44.241786 2561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:44 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967404, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:44.245174 2561 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:56:44.246766 2561 webserver.cc:460] Webserver started at http://127.21.159.254:33455/ using document root <none> and password file <none>
I20260620 14:56:44.247489 2561 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:44.247601 2561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:56:44 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967404, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:44.296988 2561 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:56:44.300453 2577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:44.301997 2561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:56:44.302203 2561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "100ddaa2de30429193b9345a76d61fcb"
format_stamp: "Formatted at 2026-06-20 14:56:40 on dist-test-slave-3mqx"
server_key: "da48cb37d7fcc2ae8ef73a9db6ed634c"
server_key_iv: "fdbca1386ad26a6963fa62fbe0f5b04f"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:44.303309 2561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:44.321472 2561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:44.325984 2561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:44.326315 2561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:44.345633 2561 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:35385
I20260620 14:56:44.345654 2628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:35385 every 8 connection(s)
I20260620 14:56:44.347303 2561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:56:44.351351 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2561
I20260620 14:56:44.352202 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:43989
--local_ip_for_outbound_sockets=127.21.159.193
--tserver_master_addrs=127.21.159.254:35385
--webserver_port=38619
--webserver_interface=127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:56:44.354691 2629 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:56:44.358105 2629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Bootstrap starting.
I20260620 14:56:44.385454 2629 log.cc:824] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:44.396590 2629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:56:44.397279 2629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Bootstrap complete.
I20260620 14:56:44.407407 2629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:44.408386 2629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100ddaa2de30429193b9345a76d61fcb, State: Initialized, Role: FOLLOWER
I20260620 14:56:44.409053 2629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:44.409304 2629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:56:44.409432 2629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:56:44.409621 2629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:56:44.413723 2629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:44.414247 2629 leader_election.cc:302] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100ddaa2de30429193b9345a76d61fcb; no voters:
I20260620 14:56:44.414942 2629 leader_election.cc:288] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:56:44.415114 2633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:56:44.416355 2633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [term 2 LEADER]: Becoming Leader. State: Replica: 100ddaa2de30429193b9345a76d61fcb, State: Running, Role: LEADER
I20260620 14:56:44.416976 2633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } }
I20260620 14:56:44.417333 2629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:56:44.419760 2635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100ddaa2de30429193b9345a76d61fcb. Latest consensus state: current_term: 2 leader_uuid: "100ddaa2de30429193b9345a76d61fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } } }
I20260620 14:56:44.420925 2635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:44.420495 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "100ddaa2de30429193b9345a76d61fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ddaa2de30429193b9345a76d61fcb" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35385 } } }
I20260620 14:56:44.421070 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:44.428692 2640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:56:44.435981 2640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:56:44.438020 2640 catalog_manager.cc:1259] Loaded cluster ID: 1edc513b5ef149c7aeda570358c8d449
I20260620 14:56:44.438163 2640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:56:44.443946 2640 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:56:44.445909 2640 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Loaded TSK: 0
I20260620 14:56:44.447992 2640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:56:44.456779 2629 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 14:56:44.849486 2631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:44.849931 2631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:44.850098 2631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:44.863538 2631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:44.863778 2631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:44.864022 2631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:56:44.881897 2631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:43989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=38619
--tserver_master_addrs=127.21.159.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:44.883778 2631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:44.885901 2631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:44.924196 2667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:44.925563 2666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:44.927400 2669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:44.935204 2631 server_base.cc:1034] running on GCE node
I20260620 14:56:44.936393 2631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:44.939390 2631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:44.940662 2631 hybrid_clock.cc:648] HybridClock initialized: now 1781967404940534 us; error 124 us; skew 500 ppm
I20260620 14:56:44.941113 2631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:44 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967404, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:44.945837 2631 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:56:44.957499 2631 webserver.cc:460] Webserver started at http://127.21.159.193:38619/ using document root <none> and password file <none>
I20260620 14:56:44.958623 2631 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:44.958802 2631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:56:44 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967404, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:45.013413 2631 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.011s sys 0.007s
I20260620 14:56:45.027731 2678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:45.033962 2631 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.008s
I20260620 14:56:45.034317 2631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "188a1fb6a1c64d43aca3a596b111f4d4"
format_stamp: "Formatted at 2026-06-20 14:56:41 on dist-test-slave-3mqx"
server_key: "4a2adfd50ae2cd1a0512b1b8e66c3ac2"
server_key_iv: "d26b035cfbf704514a693834e1270f10"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:45.036090 2631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:45.087641 2631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:45.092638 2631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:45.093056 2631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:45.094427 2631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:45.096814 2631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:45.096942 2631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:45.097064 2631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:45.097153 2631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:45.173394 2631 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:43989
I20260620 14:56:45.173455 2790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:43989 every 8 connection(s)
I20260620 14:56:45.175004 2631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:56:45.175843 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2631
I20260620 14:56:45.176944 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:46429
--local_ip_for_outbound_sockets=127.21.159.194
--tserver_master_addrs=127.21.159.254:35385
--webserver_port=43419
--webserver_interface=127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:56:45 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967404, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:45.221367 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:45.181446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:35385 (local address 127.21.159.193:58001)
0620 14:56:45.186161 (+ 4715us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:45.186221 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:45.188996 (+ 2775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:45.195756 (+ 6760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:45.195772 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:45.196535 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:45.197492 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:45.197526 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.200398 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.200407 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:45.203148 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:45.203183 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.205770 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.205781 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:45.206186 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:45.207242 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:45.207288 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:45.210622 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:45.215621 (+ 4999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:45.215633 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:45.215652 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:45.216314 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:45.216968 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:45.216976 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:45.216982 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:45.217254 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:45.218403 (+ 1149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:45.218413 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:45.218912 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 14:56:45.219287 (+ 375us) client_negotiation.cc:240] Negotiation successful
0620 14:56:45.219900 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4418,"thread_start_us":4156,"threads_started":1}
I20260620 14:56:45.221369 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:45.181057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:58001 (local address 127.21.159.254:35385)
0620 14:56:45.185485 (+ 4428us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:45.185515 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:45.186383 (+ 868us) server_negotiation.cc:407] Connection header received
0620 14:56:45.191689 (+ 5306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:45.191720 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:45.192296 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:45.192735 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:45.197806 (+ 5071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.200056 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.203417 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.204869 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.205098 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:45.210995 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:45.211009 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:45.211033 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:45.211130 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:45.215396 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:45.216503 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:45.216514 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:45.216522 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:45.216769 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:45.217433 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:45.217440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:45.217444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:45.218242 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:45.218470 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:45.219564 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0620 14:56:45.220007 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4055,"thread_start_us":6191,"threads_started":1}
I20260620 14:56:45.225921 2791 heartbeater.cc:347] Connected to a master server at 127.21.159.254:35385
I20260620 14:56:45.226280 2791 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:45.227185 2791 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
W20260620 14:56:45.230595 2791 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:35385 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:56:45.230830 2791 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:45.231158 2791 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
I20260620 14:56:45.232791 2791 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:45.233062 2791 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
W20260620 14:56:45.234277 2791 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:56:45.657752 2795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:45.658205 2795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:45.658386 2795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:45.670457 2795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:45.670630 2795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:45.670877 2795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:56:45.686214 2795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:46429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=43419
--tserver_master_addrs=127.21.159.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:45.687763 2795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:45.689256 2795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:45.707894 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:45.710573 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:45.717474 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:45.718204 2795 server_base.cc:1034] running on GCE node
I20260620 14:56:45.718938 2795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:45.720702 2795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:45.721975 2795 hybrid_clock.cc:648] HybridClock initialized: now 1781967405721909 us; error 51 us; skew 500 ppm
I20260620 14:56:45.722473 2795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:45 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967405, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:45.726354 2795 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:56:45.730947 2795 webserver.cc:460] Webserver started at http://127.21.159.194:43419/ using document root <none> and password file <none>
I20260620 14:56:45.732000 2795 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:45.732172 2795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:56:45 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967405, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
Jun 20 14:56:45 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967405, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:45.771677 2795 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 14:56:45.779771 2822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:45.787488 2795 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.003s
I20260620 14:56:45.787765 2795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "873acfa943bf4544854a0a2bb4fd517d"
format_stamp: "Formatted at 2026-06-20 14:56:42 on dist-test-slave-3mqx"
server_key: "67853cb0bb137ee40c09ae396a3d2ea5"
server_key_iv: "c5e005ecd25e0b2731e80c9a5894e5db"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:45.788973 2795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:45.810506 2795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:45.816426 2795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:45.816795 2795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:45.818282 2795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:45.820248 2795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:45.820382 2795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:45.820533 2795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:45.820647 2795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:45.881163 2795 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:46429
I20260620 14:56:45.881219 2935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:46429 every 8 connection(s)
I20260620 14:56:45.882773 2795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:56:45.883998 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2795
I20260620 14:56:45.885073 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.159.212:33583/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:34903
--local_ip_for_outbound_sockets=127.21.159.195
--tserver_master_addrs=127.21.159.254:35385
--webserver_port=43619
--webserver_interface=127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:56:45 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967405, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:45.936970 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:45.896321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:52243 (local address 127.21.159.254:35385)
0620 14:56:45.903577 (+ 7256us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:45.903588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:45.908689 (+ 5101us) server_negotiation.cc:407] Connection header received
0620 14:56:45.911891 (+ 3202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:45.911903 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:45.912037 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:45.912163 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:45.915517 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.916767 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.919826 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.920437 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.920549 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:45.928315 (+ 7766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:45.928324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:45.928329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:45.928380 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:45.932102 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:45.933187 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:45.933194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:45.933197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:45.933351 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:45.934875 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:45.934880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:45.934884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:45.935310 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:45.935446 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:45.936391 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 14:56:45.936617 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7065,"thread_start_us":6929,"threads_started":1}
I20260620 14:56:45.939044 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:45.895630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:35385 (local address 127.21.159.194:52243)
0620 14:56:45.907949 (+ 12319us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:45.908022 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:45.911371 (+ 3349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:45.913118 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:45.913137 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:45.914113 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:45.915211 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:45.915251 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.917006 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.917011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:45.919601 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:45.919620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:45.921017 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:45.921024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:45.921339 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:45.922206 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:45.922241 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:45.927936 (+ 5695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:45.932325 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:45.932336 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:45.932357 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:45.933027 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:45.934450 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:45.934455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:45.934458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:45.934718 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:45.935538 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:45.935546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:45.935950 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 14:56:45.937228 (+ 1278us) client_negotiation.cc:240] Negotiation successful
0620 14:56:45.937810 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11901,"thread_start_us":11692,"threads_started":1}
I20260620 14:56:45.940821 2936 heartbeater.cc:347] Connected to a master server at 127.21.159.254:35385
I20260620 14:56:45.941170 2936 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:45.942058 2936 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
I20260620 14:56:45.944159 2594 ts_manager.cc:194] Registered new tserver with Master: 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429)
I20260620 14:56:45.946349 2594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:52243
I20260620 14:56:46.234408 2791 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:46.234839 2791 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
I20260620 14:56:46.236267 2594 ts_manager.cc:194] Registered new tserver with Master: 188a1fb6a1c64d43aca3a596b111f4d4 (127.21.159.193:43989)
I20260620 14:56:46.237490 2594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:58001
Jun 20 14:56:46 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967405, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
W20260620 14:56:46.365262 2939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:46.365608 2939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:46.365756 2939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:46.373296 2939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:46.373457 2939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:46.373649 2939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:56:46.383865 2939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.159.212:33583/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:34903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=43619
--tserver_master_addrs=127.21.159.254:35385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:46.385267 2939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:46.386755 2939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:46.412576 2945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:46.412832 2946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:46.418498 2948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:46.419378 2939 server_base.cc:1034] running on GCE node
I20260620 14:56:46.420150 2939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:46.421386 2939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:46.422628 2939 hybrid_clock.cc:648] HybridClock initialized: now 1781967406422548 us; error 62 us; skew 500 ppm
I20260620 14:56:46.423000 2939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:46 dist-test-slave-3mqx krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967406, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:46.427407 2939 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:56:46.432961 2939 webserver.cc:460] Webserver started at http://127.21.159.195:43619/ using document root <none> and password file <none>
I20260620 14:56:46.434172 2939 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:46.434381 2939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:56:46 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967406, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:46.462436 2939 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:56:46.466207 2956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:46.468091 2939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 14:56:46.468278 2939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad"
format_stamp: "Formatted at 2026-06-20 14:56:43 on dist-test-slave-3mqx"
server_key: "2b6f5cf801364e6c45ec8b5fc9ec5f1f"
server_key_iv: "b3f214b3558cba6ae1ba5bad3afa1d9c"
server_key_version: "kuduclusterkey@0"
I20260620 14:56:46.469610 2939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:46.495496 2939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:46.503314 2939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:46.503655 2939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:46.505038 2939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:46.506857 2939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:46.506970 2939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:46.507117 2939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:46.507208 2939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:46.595175 2939 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:34903
Jun 20 14:56:46 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967405, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for HTTP/127.21.159.212@KRBTEST.COM
I20260620 14:56:46.595232 3069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:34903 every 8 connection(s)
I20260620 14:56:46.597124 2939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:56:46.606119 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 2939
Jun 20 14:56:46 dist-test-slave-3mqx krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967406, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:46.666222 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.607191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.21.159.254:35385)
0620 14:56:46.627076 (+ 19885us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.627083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.635969 (+ 8886us) server_negotiation.cc:407] Connection header received
0620 14:56:46.639887 (+ 3918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.639893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.639998 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.640117 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:46.643847 (+ 3730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.645201 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.647385 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.648083 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.648226 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.656392 (+ 8166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:46.656402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:46.656408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:46.656466 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:46.660450 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:46.663125 (+ 2675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:46.663132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:46.663136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:46.663341 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:46.663932 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:46.663939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:46.663943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:46.664411 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:46.664786 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.665478 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.665758 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19733,"thread_start_us":19288,"threads_started":1}
I20260620 14:56:46.672327 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.604549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:43941 (local address 127.21.159.254:35385)
0620 14:56:46.626630 (+ 22081us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.626639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.631033 (+ 4394us) server_negotiation.cc:407] Connection header received
0620 14:56:46.638750 (+ 7717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.638758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.638882 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.638973 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:46.642292 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.643627 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.649112 (+ 5485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.649911 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.650032 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.661924 (+ 11892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:46.661934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:46.661939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:46.661993 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:46.665543 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:46.667698 (+ 2155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:46.667704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:46.667708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:46.667859 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:46.669261 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:46.669268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:46.669271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:46.669699 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:46.669843 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.671583 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.671870 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21893,"thread_start_us":34199,"threads_started":1}
I20260620 14:56:46.674916 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.603471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:35385 (local address 127.21.159.195:43941)
0620 14:56:46.630801 (+ 27330us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:46.630871 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:46.636342 (+ 5471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:46.639696 (+ 3354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:46.639712 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:46.641110 (+ 1398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:46.642044 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.642076 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.645503 (+ 3427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.645510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.648921 (+ 3411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.648940 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.650439 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.650448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.650796 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.655419 (+ 4623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:46.655468 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:46.661282 (+ 5814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:46.665736 (+ 4454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:46.665746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:46.665767 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:46.667521 (+ 1754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:46.668777 (+ 1256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:46.668783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:46.668787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:46.669098 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:46.670441 (+ 1343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:46.670452 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:46.671001 (+ 549us) client_negotiation.cc:769] Sending connection context
0620 14:56:46.672581 (+ 1580us) client_negotiation.cc:240] Negotiation successful
0620 14:56:46.673217 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26923,"thread_start_us":27474,"threads_started":1}
I20260620 14:56:46.676678 3070 heartbeater.cc:347] Connected to a master server at 127.21.159.254:35385
I20260620 14:56:46.677011 3070 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:46.677917 3070 heartbeater.cc:511] Master 127.21.159.254:35385 requested a full tablet report, sending...
I20260620 14:56:46.680055 2594 ts_manager.cc:194] Registered new tserver with Master: 79108d4bdcfd4aa6b6c2f0d92bc3c3ad (127.21.159.195:34903)
I20260620 14:56:46.681339 2594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:43941
I20260620 14:56:46.683534 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:56:46.751277 3074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.719844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.21.159.254:35385)
0620 14:56:46.720060 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.720067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.731792 (+ 11725us) server_negotiation.cc:407] Connection header received
0620 14:56:46.732041 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.732047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.732154 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.732265 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:46.734809 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.736103 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.738356 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.739169 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.739301 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.743413 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:46.743421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:46.743427 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:46.743478 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:46.747549 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:46.748571 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:46.748579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:46.748583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:46.748746 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:46.749259 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:46.749265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:46.749269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:46.749685 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:46.749917 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.750591 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.750855 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:56:46.758750 2594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:56:46.771442 2594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:56:46.840975 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.811817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:34903 (local address 127.0.0.1:46556)
0620 14:56:46.823658 (+ 11841us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:46.823694 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:46.823891 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:46.828681 (+ 4790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:46.828694 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:46.828745 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:46.829307 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.829326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.833443 (+ 4117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.833449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.836642 (+ 3193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.836658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.839812 (+ 3154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.839819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.840103 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.840167 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:56:46.840292 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:56:46.840508 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11623,"mutex_wait_us":1,"thread_start_us":7458,"threads_started":1}
I20260620 14:56:46.842870 3090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.822250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54824)
0620 14:56:46.832186 (+ 9936us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:46.832208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:46.832327 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:46.834371 (+ 2044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:46.834376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:46.834409 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:46.834869 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.834884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.836985 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.836992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.839261 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.839277 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.841952 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.841965 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.842185 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.842219 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:56:46.842312 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:56:46.842461 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9760,"mutex_wait_us":3,"thread_start_us":10113,"threads_started":1}
I20260620 14:56:46.843683 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.812918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.21.159.195:34903)
0620 14:56:46.820912 (+ 7994us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.820923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.827853 (+ 6930us) server_negotiation.cc:407] Connection header received
0620 14:56:46.827992 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.828003 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.828332 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.828595 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:46.829596 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.833264 (+ 3668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.837792 (+ 4528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.839515 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.841337 (+ 1822us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.842882 (+ 1545us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.843033 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.843319 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7757,"thread_start_us":7568,"threads_started":1}
I20260620 14:56:46.848940 3086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.811559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:46429 (local address 127.0.0.1:56494)
0620 14:56:46.819785 (+ 8226us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:46.819855 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:46.820100 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:46.830144 (+ 10044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:46.830151 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:46.830183 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:46.830631 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.830646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.839368 (+ 8722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.839375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.845509 (+ 6134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.845529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.848019 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.848026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.848231 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.848278 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:56:46.848383 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:56:46.848539 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7879,"mutex_wait_us":2525,"thread_start_us":11670,"threads_started":1}
I20260620 14:56:46.848295 3004 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4335a787b545d7a86c099cfaaff7f3 (DEFAULT_TABLE table=test-table [id=d30fe327c638451e8c72cb7679bbe286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:56:46.849829 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.812434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.21.159.194:46429)
0620 14:56:46.821395 (+ 8961us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.821405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.821444 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:56:46.821561 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.821571 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.829241 (+ 7670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.829470 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:46.831111 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.832754 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.845773 (+ 13019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.847436 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.847605 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.847787 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.849303 (+ 1516us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.849498 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8776,"thread_start_us":10151,"threads_started":1}
I20260620 14:56:46.850061 3004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4335a787b545d7a86c099cfaaff7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:46.851709 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.822351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.21.159.193:43989)
0620 14:56:46.833020 (+ 10669us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.833092 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.833134 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:56:46.833578 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.833592 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.833890 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.834099 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:46.835133 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.836828 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.839464 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.841073 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.841570 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.841768 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.851128 (+ 9360us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.851327 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10478,"thread_start_us":16705,"threads_started":1}
I20260620 14:56:46.854091 2871 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4335a787b545d7a86c099cfaaff7f3 (DEFAULT_TABLE table=test-table [id=d30fe327c638451e8c72cb7679bbe286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:56:46.855786 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4335a787b545d7a86c099cfaaff7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:46.855950 2726 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4335a787b545d7a86c099cfaaff7f3 (DEFAULT_TABLE table=test-table [id=d30fe327c638451e8c72cb7679bbe286]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:56:46.857563 2726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4335a787b545d7a86c099cfaaff7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:46.880673 3091 tablet_bootstrap.cc:492] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Bootstrap starting.
I20260620 14:56:46.880661 3092 tablet_bootstrap.cc:492] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4: Bootstrap starting.
I20260620 14:56:46.886053 3091 tablet_bootstrap.cc:654] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:46.887315 3092 tablet_bootstrap.cc:654] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:46.887573 3091 log.cc:824] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:46.888787 3092 log.cc:824] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:46.902639 3091 tablet_bootstrap.cc:492] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: No bootstrap required, opened a new log
I20260620 14:56:46.902985 3091 ts_tablet_manager.cc:1397] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.010s
I20260620 14:56:46.903301 3092 tablet_bootstrap.cc:492] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4: No bootstrap required, opened a new log
I20260620 14:56:46.903586 3092 ts_tablet_manager.cc:1397] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.016s
I20260620 14:56:46.915637 3092 raft_consensus.cc:348] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:46.916177 3092 raft_consensus.cc:374] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:46.916277 3092 raft_consensus.cc:729] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 188a1fb6a1c64d43aca3a596b111f4d4, State: Initialized, Role: FOLLOWER
I20260620 14:56:46.916939 3091 raft_consensus.cc:348] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:46.917079 3092 consensus_queue.cc:260] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:46.917392 3091 raft_consensus.cc:374] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:46.917495 3091 raft_consensus.cc:729] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79108d4bdcfd4aa6b6c2f0d92bc3c3ad, State: Initialized, Role: FOLLOWER
I20260620 14:56:46.918183 3091 consensus_queue.cc:260] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:46.918673 3095 tablet_bootstrap.cc:492] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Bootstrap starting.
I20260620 14:56:46.937072 2791 heartbeater.cc:503] Master 127.21.159.254:35385 was elected leader, sending a full tablet report...
I20260620 14:56:46.944092 3098 raft_consensus.cc:482] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:56:46.944527 3098 raft_consensus.cc:504] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:46.949918 3095 tablet_bootstrap.cc:654] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:46.951099 2936 heartbeater.cc:503] Master 127.21.159.254:35385 was elected leader, sending a full tablet report...
I20260620 14:56:46.956816 3092 ts_tablet_manager.cc:1428] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4: Time spent starting tablet: real 0.053s user 0.012s sys 0.006s
I20260620 14:56:46.961086 3095 log.cc:824] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:46.963112 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.947687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:47129 (local address 127.21.159.194:46429)
0620 14:56:46.948006 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.948016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.948689 (+ 673us) server_negotiation.cc:407] Connection header received
0620 14:56:46.949344 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.949350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.949433 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.949512 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:46.951095 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.952336 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.959568 (+ 7232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.961039 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.961203 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.961286 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.962629 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.962795 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:56:46.964085 3098 leader_election.cc:288] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429), 79108d4bdcfd4aa6b6c2f0d92bc3c3ad (127.21.159.195:34903)
I20260620 14:56:46.964869 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.947500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:46429 (local address 127.21.159.193:47129)
0620 14:56:46.948551 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:46.948577 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:46.948981 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:46.949779 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:46.949786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:46.949822 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:46.950230 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.950245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.952562 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.952567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.957670 (+ 5103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.957688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.961592 (+ 3904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.961599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.961850 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.961883 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:56:46.964309 (+ 2426us) client_negotiation.cc:240] Negotiation successful
0620 14:56:46.964503 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":709,"threads_started":1}
I20260620 14:56:46.965126 3070 heartbeater.cc:503] Master 127.21.159.254:35385 was elected leader, sending a full tablet report...
I20260620 14:56:46.967397 3091 ts_tablet_manager.cc:1428] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Time spent starting tablet: real 0.064s user 0.016s sys 0.030s
I20260620 14:56:46.967361 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4335a787b545d7a86c099cfaaff7f3" candidate_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873acfa943bf4544854a0a2bb4fd517d" is_pre_election: true
I20260620 14:56:46.972625 3095 tablet_bootstrap.cc:492] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: No bootstrap required, opened a new log
I20260620 14:56:46.972965 3095 ts_tablet_manager.cc:1397] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Time spent bootstrapping tablet: real 0.055s user 0.014s sys 0.000s
W20260620 14:56:46.973500 2680 leader_election.cc:341] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429): Illegal state: must be running to vote when last-logged opid is not known
I20260620 14:56:46.976471 3097 raft_consensus.cc:482] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:56:46.976884 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.962953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:40057 (local address 127.21.159.195:34903)
0620 14:56:46.963506 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.963516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.964206 (+ 690us) server_negotiation.cc:407] Connection header received
0620 14:56:46.968224 (+ 4018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.968234 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.968337 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.968422 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:46.970423 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.971555 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.974200 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.975577 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.975733 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.975814 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:46.976417 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 14:56:46.976580 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260620 14:56:46.976773 3097 raft_consensus.cc:504] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:46.977681 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.962442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:34903 (local address 127.21.159.193:40057)
0620 14:56:46.964099 (+ 1657us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:46.964122 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:46.967881 (+ 3759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:46.968691 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:46.968699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:46.968735 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:46.969136 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.969151 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.971783 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.971789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.974004 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.974021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.976023 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.976030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.976216 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.976250 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:56:46.977125 (+ 875us) client_negotiation.cc:240] Negotiation successful
0620 14:56:46.977287 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"thread_start_us":1349,"threads_started":1}
I20260620 14:56:46.978852 3097 leader_election.cc:288] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429), 188a1fb6a1c64d43aca3a596b111f4d4 (127.21.159.193:43989)
I20260620 14:56:46.980013 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4335a787b545d7a86c099cfaaff7f3" candidate_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" is_pre_election: true
I20260620 14:56:46.980635 3024 raft_consensus.cc:2454] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 188a1fb6a1c64d43aca3a596b111f4d4 in term 0.
I20260620 14:56:46.982514 2682 leader_election.cc:302] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 188a1fb6a1c64d43aca3a596b111f4d4, 79108d4bdcfd4aa6b6c2f0d92bc3c3ad; no voters: 873acfa943bf4544854a0a2bb4fd517d
I20260620 14:56:46.983843 3098 raft_consensus.cc:2781] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:56:46.984050 3098 raft_consensus.cc:482] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:56:46.984205 3098 raft_consensus.cc:3037] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:47.000029 3098 raft_consensus.cc:504] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:47.001801 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4335a787b545d7a86c099cfaaff7f3" candidate_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873acfa943bf4544854a0a2bb4fd517d"
W20260620 14:56:47.002562 2680 leader_election.cc:341] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429): Illegal state: must be running to vote when last-logged opid is not known
I20260620 14:56:47.003000 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.979323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.21.159.195:57175)
0620 14:56:46.979502 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:46.979545 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:46.979675 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:46.988310 (+ 8635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:46.988320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:46.988371 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:46.988696 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.988715 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.990829 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.990839 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:46.995586 (+ 4747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:46.995614 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.001712 (+ 6098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.001725 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:47.002088 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:47.002169 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 14:56:47.002309 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 14:56:47.002553 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:56:47.004015 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.981280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:57175 (local address 127.21.159.193:43989)
0620 14:56:46.981548 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.981558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:46.981585 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:56:46.981667 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:46.981676 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:46.981780 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:46.981908 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:46.988966 (+ 7058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.990588 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.995891 (+ 5303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:46.998388 (+ 2497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:46.998623 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:46.998750 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:47.003422 (+ 4672us) server_negotiation.cc:299] Negotiation successful
0620 14:56:47.003661 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:56:47.005095 2746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4335a787b545d7a86c099cfaaff7f3" candidate_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" is_pre_election: true
I20260620 14:56:47.005715 2746 raft_consensus.cc:2379] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 79108d4bdcfd4aa6b6c2f0d92bc3c3ad in current term 1: Already voted for candidate 188a1fb6a1c64d43aca3a596b111f4d4 in this term.
I20260620 14:56:47.006307 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4335a787b545d7a86c099cfaaff7f3" candidate_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad"
I20260620 14:56:47.006660 3024 raft_consensus.cc:3037] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:47.011785 3024 raft_consensus.cc:2454] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 188a1fb6a1c64d43aca3a596b111f4d4 in term 1.
I20260620 14:56:47.012544 2682 leader_election.cc:302] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 188a1fb6a1c64d43aca3a596b111f4d4, 79108d4bdcfd4aa6b6c2f0d92bc3c3ad; no voters: 873acfa943bf4544854a0a2bb4fd517d
I20260620 14:56:47.013125 3096 raft_consensus.cc:2781] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:56:47.017707 3098 leader_election.cc:288] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [CANDIDATE]: Term 1 election: Requested vote from peers 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429), 79108d4bdcfd4aa6b6c2f0d92bc3c3ad (127.21.159.195:34903)
I20260620 14:56:47.019390 3096 raft_consensus.cc:686] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [term 1 LEADER]: Becoming Leader. State: Replica: 188a1fb6a1c64d43aca3a596b111f4d4, State: Running, Role: LEADER
I20260620 14:56:47.020299 3096 consensus_queue.cc:237] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:47.040705 3072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.979658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:46429 (local address 127.21.159.195:54567)
0620 14:56:47.004986 (+ 25328us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.005016 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:47.005188 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:47.018251 (+ 13063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:47.018261 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:47.018313 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:47.018617 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:47.024192 (+ 5575us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.026740 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.026750 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:47.035349 (+ 8599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:47.035386 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.038679 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.038691 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:47.039933 (+ 1242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:47.039990 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:56:47.040157 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:56:47.040332 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25135,"threads_started":1}
I20260620 14:56:47.041157 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:46.979278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:54567 (local address 127.21.159.194:46429)
0620 14:56:46.979462 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:46.979470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:47.005109 (+ 25639us) server_negotiation.cc:407] Connection header received
0620 14:56:47.017091 (+ 11982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:47.017102 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:47.017255 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:47.017358 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:47.024745 (+ 7387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.026246 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.035695 (+ 9449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.038043 (+ 2348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.038294 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:47.038421 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:47.040252 (+ 1831us) server_negotiation.cc:299] Negotiation successful
0620 14:56:47.040499 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:56:47.043013 3095 raft_consensus.cc:348] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:47.043666 3095 raft_consensus.cc:374] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:47.043825 3095 raft_consensus.cc:729] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873acfa943bf4544854a0a2bb4fd517d, State: Initialized, Role: FOLLOWER
I20260620 14:56:47.044698 3095 consensus_queue.cc:260] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:56:47.045759 2594 catalog_manager.cc:5495] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 188a1fb6a1c64d43aca3a596b111f4d4 (127.21.159.193). New cstate: current_term: 1 leader_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "188a1fb6a1c64d43aca3a596b111f4d4" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } health_report { overall_health: HEALTHY } } }
I20260620 14:56:47.046379 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4335a787b545d7a86c099cfaaff7f3" candidate_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "873acfa943bf4544854a0a2bb4fd517d" is_pre_election: true
I20260620 14:56:47.046753 2891 raft_consensus.cc:2454] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79108d4bdcfd4aa6b6c2f0d92bc3c3ad in term 0.
I20260620 14:56:47.047379 2958 leader_election.cc:302] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79108d4bdcfd4aa6b6c2f0d92bc3c3ad, 873acfa943bf4544854a0a2bb4fd517d; no voters: 188a1fb6a1c64d43aca3a596b111f4d4
I20260620 14:56:47.049643 3095 ts_tablet_manager.cc:1428] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Time spent starting tablet: real 0.077s user 0.012s sys 0.005s
I20260620 14:56:47.050874 3097 raft_consensus.cc:2741] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 14:56:47.085175 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.079149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.21.159.193:43989)
0620 14:56:47.079332 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:47.079338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:47.079455 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:56:47.079726 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:47.079731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:47.079828 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:47.079987 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:56:47.080392 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.081164 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.082621 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.083118 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.083298 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:47.083896 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:56:47.084523 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:56:47.084707 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:47.084757 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 14:56:47.084861 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:56:47.094981 2891 raft_consensus.cc:3037] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:47.095279 3024 raft_consensus.cc:1261] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 1 FOLLOWER]: Refusing update from remote peer 188a1fb6a1c64d43aca3a596b111f4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:56:47.096268 3096 consensus_queue.cc:1035] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79108d4bdcfd4aa6b6c2f0d92bc3c3ad" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:56:47.098148 2891 raft_consensus.cc:1261] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 1 FOLLOWER]: Refusing update from remote peer 188a1fb6a1c64d43aca3a596b111f4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:56:47.099102 3096 consensus_queue.cc:1035] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "873acfa943bf4544854a0a2bb4fd517d" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 46429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:56:47.107661 3071 tablet.cc:2369] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:56:47.139957 2593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:56:47.140309 2593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:56:47.145531 2593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 100ddaa2de30429193b9345a76d61fcb: Sending DeleteTablet for 3 replicas of tablet 7e4335a787b545d7a86c099cfaaff7f3
I20260620 14:56:47.146775 2871 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4335a787b545d7a86c099cfaaff7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:56:47 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56494
I20260620 14:56:47.147243 3004 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4335a787b545d7a86c099cfaaff7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:56:47 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:46556
I20260620 14:56:47.148164 2726 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4335a787b545d7a86c099cfaaff7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:56:47 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54824
I20260620 14:56:47.156515 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2631
I20260620 14:56:47.157227 3124 tablet_replica.cc:330] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: stopping tablet replica
I20260620 14:56:47.158238 3124 raft_consensus.cc:2229] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:56:47.159571 3124 raft_consensus.cc:2258] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:56:47.161870 3123 tablet_replica.cc:330] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: stopping tablet replica
I20260620 14:56:47.162544 3123 raft_consensus.cc:2229] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:56:47.163199 3123 raft_consensus.cc:2258] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:56:47.165128 3124 ts_tablet_manager.cc:1905] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:56:47.173003 3124 ts_tablet_manager.cc:1918] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:56:47.175215 3123 ts_tablet_manager.cc:1905] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:56:47.182375 3123 ts_tablet_manager.cc:1918] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:56:47.182689 3124 log.cc:1196] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/7e4335a787b545d7a86c099cfaaff7f3
I20260620 14:56:47.183447 3124 ts_tablet_manager.cc:1939] T 7e4335a787b545d7a86c099cfaaff7f3 P 79108d4bdcfd4aa6b6c2f0d92bc3c3ad: Deleting consensus metadata
I20260620 14:56:47.185611 2581 catalog_manager.cc:4841] TS 79108d4bdcfd4aa6b6c2f0d92bc3c3ad (127.21.159.195:34903): tablet 7e4335a787b545d7a86c099cfaaff7f3 (table test-table [id=d30fe327c638451e8c72cb7679bbe286]) successfully deleted
I20260620 14:56:47.186121 3123 log.cc:1196] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/7e4335a787b545d7a86c099cfaaff7f3
I20260620 14:56:47.186794 3123 ts_tablet_manager.cc:1939] T 7e4335a787b545d7a86c099cfaaff7f3 P 873acfa943bf4544854a0a2bb4fd517d: Deleting consensus metadata
I20260620 14:56:47.188690 2579 catalog_manager.cc:4841] TS 873acfa943bf4544854a0a2bb4fd517d (127.21.159.194:46429): tablet 7e4335a787b545d7a86c099cfaaff7f3 (table test-table [id=d30fe327c638451e8c72cb7679bbe286]) successfully deleted
I20260620 14:56:47.189661 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.160306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.21.159.193:43989)
0620 14:56:47.160505 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:47.160512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:47.160531 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:56:47.160603 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:47.160611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:47.160701 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:47.160819 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:47.163275 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.164451 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.166272 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.166951 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.167137 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:47.170837 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:47.170855 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:47.170876 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:47.170929 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:47.185087 (+ 14158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:47.186155 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:47.186165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:47.186169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:47.186332 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:47.186868 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:47.186874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:47.186877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:47.187433 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:47.187745 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:47.189075 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0620 14:56:47.189318 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:56:47.190346 2786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:47.322584 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2631
I20260620 14:56:47.322562 3125 tablet_replica.cc:330] T 7e4335a787b545d7a86c099cfaaff7f3 P 188a1fb6a1c64d43aca3a596b111f4d4: stopping tablet replica
W20260620 14:56:47.346908 2580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.159.193:43989 (error 108)
I20260620 14:56:47.347743 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2795
I20260620 14:56:47.367220 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.349267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.21.159.194:46429)
0620 14:56:47.349435 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:47.349440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:47.349458 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:56:47.349520 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:47.349525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:47.349612 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:47.349703 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:47.351618 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.352729 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.354621 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.355292 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.355414 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:47.358913 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:47.358933 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:47.358953 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:47.358995 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:47.363033 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:47.364028 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:47.364038 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:47.364042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:47.364207 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:47.364649 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:47.364655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:47.364660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:47.365082 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:47.365369 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:47.366593 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0620 14:56:47.366828 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:56:47.368202 2931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:47.380437 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.376277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54850)
0620 14:56:47.379391 (+ 3114us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.379965 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2918,"thread_start_us":2827,"threads_started":1}
W20260620 14:56:47.380723 2580 catalog_manager.cc:4584] TS 188a1fb6a1c64d43aca3a596b111f4d4 (127.21.159.193:43989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e4335a787b545d7a86c099cfaaff7f3: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
I20260620 14:56:47.434754 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.434135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54854)
0620 14:56:47.434398 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.434561 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 14:56:47.436061 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.435585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54868)
0620 14:56:47.435763 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.435855 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:56:47.486750 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2795
I20260620 14:56:47.496498 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.496053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54878)
0620 14:56:47.496262 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.496347 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:56:47.497713 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.497274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54886)
0620 14:56:47.497431 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.497544 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:56:47.504247 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2939
I20260620 14:56:47.518841 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.505150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.21.159.195:34903)
0620 14:56:47.505663 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:47.505668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:47.505687 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:56:47.505746 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:47.505750 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:47.505811 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:47.505912 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:47.507736 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.508543 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.509920 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:47.510303 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:47.510397 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:47.513272 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:47.513305 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:47.513320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:47.513358 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:47.516149 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:47.516852 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:47.516858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:47.516861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:47.517002 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:47.517418 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:47.517421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:47.517423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:47.517766 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:47.517980 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:47.518338 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:56:47.518498 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":230,"threads_started":1}
I20260620 14:56:47.519601 3065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:47.578605 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.578093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54888)
0620 14:56:47.578352 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.578449 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:56:47.579734 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:47.579332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:54892)
0620 14:56:47.579547 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:47.579613 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:43989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 14:56:47.614539 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2939
I20260620 14:56:47.631901 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 2561
I20260620 14:56:47.633165 2624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:47.747722 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 2561
I20260620 14:56:47.772112 22143 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:56:48.107991 22143 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:56:48.108197 22143 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:56:48.454704 22143 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:56:48.455 UTC [1613] LOG: received smart shutdown request
2026-06-20 14:56:48.458 UTC [1613] LOG: background worker "logical replication launcher" (PID 1622) exited with exit code 1
2026-06-20 14:56:48.458 UTC [1617] LOG: shutting down
2026-06-20 14:56:48.636 UTC [1613] LOG: database system is shut down
2026-06-20T14:56:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54490 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:56:48 dist-test-slave-3mqx krb5kdc[3141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:56:48 dist-test-slave-3mqx krb5kdc[3141](info): set up 2 sockets
Jun 20 14:56:48 dist-test-slave-3mqx krb5kdc[3141](info): commencing operation
krb5kdc: starting...
W20260620 14:56:49.955592 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:56:50 dist-test-slave-3mqx krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:56:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:56:50.194300 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36141
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:36141,127.21.159.253:37749,127.21.159.252:46171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:50.545308 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:50.545639 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:50.545816 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:50.553840 3154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:56:50.553987 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:50.554090 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:50.554188 3154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:56:50.554271 3154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:56:50.565126 3154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:36141,127.21.159.253:37749,127.21.159.252:46171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:50.566708 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:50.568239 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:50.582926 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:50.582929 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:50.582818 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:50.584051 3154 server_base.cc:1034] running on GCE node
I20260620 14:56:50.585048 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:50.586747 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:50.588020 3154 hybrid_clock.cc:648] HybridClock initialized: now 1781967410587936 us; error 78 us; skew 500 ppm
I20260620 14:56:50.588450 3154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:50 dist-test-slave-3mqx krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:50.592326 3154 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:56:50.594055 3154 webserver.cc:460] Webserver started at http://127.21.159.254:40667/ using document root <none> and password file <none>
I20260620 14:56:50.594851 3154 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:50.595008 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:50.595427 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:50.599151 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "f02b1fa0b6804afcbf1a1f8547e05a70"
format_stamp: "Formatted at 2026-06-20 14:56:50 on dist-test-slave-3mqx"
server_key: "57cece68ca1e6654e5e56c02246c8a4c"
server_key_iv: "a302bdefda42c40710ce8776bda19510"
server_key_version: "encryptionkey@0"
I20260620 14:56:50.600129 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "f02b1fa0b6804afcbf1a1f8547e05a70"
format_stamp: "Formatted at 2026-06-20 14:56:50 on dist-test-slave-3mqx"
server_key: "57cece68ca1e6654e5e56c02246c8a4c"
server_key_iv: "a302bdefda42c40710ce8776bda19510"
server_key_version: "encryptionkey@0"
I20260620 14:56:50.606127 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260620 14:56:50.610502 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:50.612300 3154 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:56:50.612547 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "f02b1fa0b6804afcbf1a1f8547e05a70"
format_stamp: "Formatted at 2026-06-20 14:56:50 on dist-test-slave-3mqx"
server_key: "57cece68ca1e6654e5e56c02246c8a4c"
server_key_iv: "a302bdefda42c40710ce8776bda19510"
server_key_version: "encryptionkey@0"
I20260620 14:56:50.612886 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:50.644050 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:50.649526 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:50.649879 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:50.671706 3154 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:36141
I20260620 14:56:50.671730 3220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:36141 every 8 connection(s)
I20260620 14:56:50.673327 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:56:50.677582 3221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:50.679014 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3154
I20260620 14:56:50.679335 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:56:50.680096 22143 external_mini_cluster.cc:1428] Setting key 7de4e442e0344c7ecfcf46280e46a066
I20260620 14:56:50.684832 3221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.159.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.253@KRBTEST.COM" created.
Jun 20 14:56:50 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:50.715716 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.688428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.21.159.254:36141)
0620 14:56:50.690358 (+ 1930us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:50.690368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:50.690436 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:56:50.692291 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:50.692298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:50.692570 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:50.692802 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:50.694255 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:50.696218 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:50.697891 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:50.698545 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:50.698767 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:50.705532 (+ 6765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:50.705547 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:50.705571 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:50.705623 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:50.710307 (+ 4684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:50.711468 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:50.711480 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:50.711487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:50.711722 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:50.712168 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:50.712173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:50.712176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:50.712742 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:50.712968 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:50.713567 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 14:56:50.714009 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":1695,"thread_start_us":1444,"threads_started":1}
I20260620 14:56:50.715716 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.688428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36141 (local address 127.0.0.1:48670)
0620 14:56:50.689423 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.689481 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:50.691848 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:50.692815 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:50.692823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:50.693658 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:50.694030 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:50.694050 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:50.696383 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:50.696389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:50.697739 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:50.697752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:50.698813 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:50.698821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:50.699033 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:50.699851 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:50.699876 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:50.705175 (+ 5299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:50.710537 (+ 5362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:50.710555 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:50.710572 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:50.711286 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:50.711865 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:50.711871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:50.711875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:50.712045 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:50.712882 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:50.712895 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:50.713126 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 14:56:50.713416 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 14:56:50.714127 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":180,"thread_start_us":448,"threads_started":1}
I20260620 14:56:50.719465 3221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:56:50.720571 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.720146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48884)
0620 14:56:50.720323 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.720446 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:56:50.720937 3172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
I20260620 14:56:50.722669 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.722204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48900)
0620 14:56:50.722366 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.722479 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:56:50.722924 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:37749: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab.
Entry for principal kudu/127.21.159.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab.
I20260620 14:56:50.736685 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:37749
--webserver_interface=127.21.159.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab
--principal=kudu/127.21.159.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:36141,127.21.159.253:37749,127.21.159.252:46171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:56:50.773241 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } attempt: 1
I20260620 14:56:50.774550 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.774062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48902)
0620 14:56:50.774266 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.774396 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 14:56:50.775687 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.775283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48912)
0620 14:56:50.775445 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.775530 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 14:56:50.775949 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:37749: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
I20260620 14:56:50.844244 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } attempt: 2
I20260620 14:56:50.845669 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.845055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48922)
0620 14:56:50.845372 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.845489 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 14:56:50.846911 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.846431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48924)
0620 14:56:50.846639 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.846726 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 14:56:50.847163 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:37749: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
I20260620 14:56:50.938462 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } attempt: 3
I20260620 14:56:50.939829 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.939313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48926)
0620 14:56:50.939543 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.939669 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:56:50.941048 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:50.940611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48930)
0620 14:56:50.940776 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:50.940865 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:56:50.941303 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:37749: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
W20260620 14:56:51.078354 3228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:51.078649 3228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:51.078809 3228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:56:51.084606 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } attempt: 4
I20260620 14:56:51.085954 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.085319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48932)
0620 14:56:51.085667 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.085774 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260620 14:56:51.086649 3228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:56:51.086804 3228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:51.086887 3228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:51.086979 3228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:56:51.087072 3228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:56:51.087138 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.086642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48938)
0620 14:56:51.086837 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.086952 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:56:51.087405 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:37749: Network error: Client connection negotiation failed: client connection to 127.21.159.253:37749: connect: Connection refused (error 111)
I20260620 14:56:51.097631 3228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:36141,127.21.159.253:37749,127.21.159.252:46171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab
--principal=kudu/127.21.159.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:37749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.159.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:51.099190 3228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:51.100543 3228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:51.112946 3233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:51.113018 3234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:51.112915 3236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:51.113826 3228 server_base.cc:1034] running on GCE node
I20260620 14:56:51.114744 3228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:51.116432 3228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:51.117661 3228 hybrid_clock.cc:648] HybridClock initialized: now 1781967411117612 us; error 50 us; skew 500 ppm
I20260620 14:56:51.118022 3228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:51.121528 3228 init.cc:377] Logged in from keytab as kudu/127.21.159.253@KRBTEST.COM (short username kudu)
I20260620 14:56:51.123178 3228 webserver.cc:460] Webserver started at http://127.21.159.253:36311/ using document root <none> and password file <none>
I20260620 14:56:51.123919 3228 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:51.124073 3228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:51.124392 3228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:51.127530 3228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data/instance:
uuid: "23e95847e2be48d8b30a6936bd962612"
format_stamp: "Formatted at 2026-06-20 14:56:51 on dist-test-slave-3mqx"
server_key: "a88bd9903066f9116e99b3d569a27b1e"
server_key_iv: "80018a6f4e7cdffebd1565b06ba7724e"
server_key_version: "encryptionkey@0"
I20260620 14:56:51.128356 3228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/wal/instance:
uuid: "23e95847e2be48d8b30a6936bd962612"
format_stamp: "Formatted at 2026-06-20 14:56:51 on dist-test-slave-3mqx"
server_key: "a88bd9903066f9116e99b3d569a27b1e"
server_key_iv: "80018a6f4e7cdffebd1565b06ba7724e"
server_key_version: "encryptionkey@0"
I20260620 14:56:51.133981 3228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:56:51.137881 3243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:51.139534 3228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:56:51.139734 3228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/wal
uuid: "23e95847e2be48d8b30a6936bd962612"
format_stamp: "Formatted at 2026-06-20 14:56:51 on dist-test-slave-3mqx"
server_key: "a88bd9903066f9116e99b3d569a27b1e"
server_key_iv: "80018a6f4e7cdffebd1565b06ba7724e"
server_key_version: "encryptionkey@0"
I20260620 14:56:51.139945 3228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:51.175833 3228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:51.180347 3228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:51.180663 3228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:51.200548 3228 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.253:37749
I20260620 14:56:51.200580 3294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.253:37749 every 8 connection(s)
I20260620 14:56:51.202023 3228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
I20260620 14:56:51.205983 3295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:51.210556 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3228
I20260620 14:56:51.210757 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-1/wal/instance
I20260620 14:56:51.211344 22143 external_mini_cluster.cc:1428] Setting key 82a1f3ba1a4cd33b44b399ff43885134
I20260620 14:56:51.213863 3295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.252@KRBTEST.COM" created.
I20260620 14:56:51.239602 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.217537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.21.159.254:36141)
0620 14:56:51.218095 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.218104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.219149 (+ 1045us) server_negotiation.cc:407] Connection header received
0620 14:56:51.221812 (+ 2663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.221817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.221888 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.221991 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.224227 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.225192 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.227761 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.228182 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.228283 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.232790 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.232798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.232805 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.232862 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.236173 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.237022 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.237026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.237028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.237153 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.237686 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.237689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.237691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.237970 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.238092 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.239136 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.239319 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":253,"threads_started":1}
I20260620 14:56:51.241182 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.218115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36141 (local address 127.0.0.1:48686)
0620 14:56:51.218918 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.218999 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.221473 (+ 2474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.222254 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.222268 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.223111 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.223940 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.223985 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.225392 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.225401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.227587 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.227605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.228430 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.228440 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.228781 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.229614 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.229651 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.232490 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.236325 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.236335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.236349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.236889 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.237295 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.237300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.237304 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.237568 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.238106 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.238115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.238645 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.238975 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.239630 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":345,"threads_started":1}
I20260620 14:56:51.243192 3295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:56:51.260080 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.243866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48942)
0620 14:56:51.244062 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.244084 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.244209 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.245342 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.245350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.245907 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.246116 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.246132 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.248311 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.248315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.249159 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.249170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.250465 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.250478 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.250652 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.251346 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.251364 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.253632 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.257495 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.257500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.257505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.257891 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.258255 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.258261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.258267 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.258414 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.259013 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.259019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.259195 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.259333 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.259608 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":1}
I20260620 14:56:51.260114 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.244167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.21.159.253:37749)
0620 14:56:51.244798 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.244809 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.244843 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:56:51.244906 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.244917 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.245172 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.245316 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.246326 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.248158 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.249307 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.250311 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.250451 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.253882 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.253900 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.253922 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.253966 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.257335 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.258025 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.258038 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.258040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.258148 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.258526 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.258530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.258533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.258892 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.259138 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.259487 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.259694 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":427,"thread_start_us":287,"threads_started":1}
I20260620 14:56:51.262683 3295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:56:51.264078 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.263597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:47976)
0620 14:56:51.263807 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.263917 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 14:56:51.264539 3247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
I20260620 14:56:51.266115 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.265725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:47978)
0620 14:56:51.265939 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.266015 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:56:51.266479 3295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab.
Entry for principal kudu/127.21.159.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab.
I20260620 14:56:51.268923 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:46171
--webserver_interface=127.21.159.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab
--principal=kudu/127.21.159.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:36141,127.21.159.253:37749,127.21.159.252:46171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:56:51.315797 3295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 1
I20260620 14:56:51.317184 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.316501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:47984)
0620 14:56:51.316822 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.316975 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 14:56:51.318395 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.317964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:47986)
0620 14:56:51.318138 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.318236 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:56:51.318660 3295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
I20260620 14:56:51.386699 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } attempt: 5
I20260620 14:56:51.386896 3295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 2
I20260620 14:56:51.388350 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.387813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:47990)
0620 14:56:51.388049 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.388144 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 14:56:51.389431 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.389034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:47996)
0620 14:56:51.389187 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.389275 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:56:51.389801 3295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:56:51.402087 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.387569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48946)
0620 14:56:51.387934 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.387957 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.388101 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.388651 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.388656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.389164 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.389356 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.389367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.390631 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.390637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.391603 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.391615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.392632 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.392637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.392809 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.393511 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.393529 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.395984 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.399676 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.399680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.399683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.400080 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.400468 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.400472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.400474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.400610 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.401322 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.401325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.401463 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.401584 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.401824 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260620 14:56:51.402478 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.387791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.21.159.253:37749)
0620 14:56:51.387966 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.387972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.388025 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:56:51.388241 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.388246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.388348 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.388511 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.389518 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.390497 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.391778 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.392448 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.392592 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.396206 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.396213 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.396218 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.396260 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.399503 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.400200 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.400206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.400209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.400358 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.400737 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.400744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.400749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.401202 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.401381 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.401748 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.402067 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:56:51.403609 3221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:56:51.404666 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.404308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48012)
0620 14:56:51.404468 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.404544 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:56:51.405611 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.405279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48016)
0620 14:56:51.405415 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.405507 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:56:51.405828 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
I20260620 14:56:51.457135 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 1
I20260620 14:56:51.458537 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.457943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48018)
0620 14:56:51.458197 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.458333 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 14:56:51.459784 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.459347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48022)
0620 14:56:51.459537 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.459635 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:56:51.460052 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
I20260620 14:56:51.481066 3295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 3
I20260620 14:56:51.482391 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.481853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48034)
0620 14:56:51.482077 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.482190 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:56:51.483595 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.483140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48042)
0620 14:56:51.483376 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.483458 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260620 14:56:51.483815 3295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
I20260620 14:56:51.528313 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 2
I20260620 14:56:51.529727 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.529086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48056)
0620 14:56:51.529364 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.529489 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 14:56:51.530970 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.530544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48072)
0620 14:56:51.530740 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.530836 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:56:51.531250 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
W20260620 14:56:51.621608 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:51.622013 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:51.622174 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:56:51.627072 3295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 4
I20260620 14:56:51.628330 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.627761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48076)
0620 14:56:51.628025 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.628133 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 14:56:51.629351 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.628996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48090)
0620 14:56:51.629143 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.629225 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:56:51.629578 3295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
W20260620 14:56:51.630656 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:56:51.630779 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:51.630874 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:51.630972 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:56:51.631052 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:56:51.637544 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 3
I20260620 14:56:51.638737 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.638204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48098)
0620 14:56:51.638432 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.638543 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 14:56:51.639868 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.639439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48100)
0620 14:56:51.639643 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.639734 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260620 14:56:51.640105 3221 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:46171: Network error: Client connection negotiation failed: client connection to 127.21.159.252:46171: connect: Connection refused (error 111)
I20260620 14:56:51.642622 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:36141,127.21.159.253:37749,127.21.159.252:46171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab
--principal=kudu/127.21.159.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:46171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.159.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:51.644105 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:51.645496 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:51.657179 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:51.657101 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:51.657681 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:51.658308 3303 server_base.cc:1034] running on GCE node
I20260620 14:56:51.659236 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:51.660796 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:51.662050 3303 hybrid_clock.cc:648] HybridClock initialized: now 1781967411661972 us; error 75 us; skew 500 ppm
I20260620 14:56:51.662425 3303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:51.665704 3303 init.cc:377] Logged in from keytab as kudu/127.21.159.252@KRBTEST.COM (short username kudu)
I20260620 14:56:51.667259 3303 webserver.cc:460] Webserver started at http://127.21.159.252:44109/ using document root <none> and password file <none>
I20260620 14:56:51.667932 3303 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:51.668031 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:51.668282 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:51.671495 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data/instance:
uuid: "9b9920cc03284930bc32f55f26a7fd40"
format_stamp: "Formatted at 2026-06-20 14:56:51 on dist-test-slave-3mqx"
server_key: "0a7e847cf5ee94615bff9c94cfd5b9c5"
server_key_iv: "61ac103a04c7c5dafeb4714a9bd3aa39"
server_key_version: "encryptionkey@0"
I20260620 14:56:51.672370 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/wal/instance:
uuid: "9b9920cc03284930bc32f55f26a7fd40"
format_stamp: "Formatted at 2026-06-20 14:56:51 on dist-test-slave-3mqx"
server_key: "0a7e847cf5ee94615bff9c94cfd5b9c5"
server_key_iv: "61ac103a04c7c5dafeb4714a9bd3aa39"
server_key_version: "encryptionkey@0"
I20260620 14:56:51.677752 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:56:51.681490 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:51.682957 3303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:56:51.683198 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/wal
uuid: "9b9920cc03284930bc32f55f26a7fd40"
format_stamp: "Formatted at 2026-06-20 14:56:51 on dist-test-slave-3mqx"
server_key: "0a7e847cf5ee94615bff9c94cfd5b9c5"
server_key_iv: "61ac103a04c7c5dafeb4714a9bd3aa39"
server_key_version: "encryptionkey@0"
I20260620 14:56:51.683465 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:51.697077 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:51.701630 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:51.701967 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:51.721468 3303 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.252:46171
I20260620 14:56:51.721504 3369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.252:46171 every 8 connection(s)
I20260620 14:56:51.723042 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
I20260620 14:56:51.727075 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:51.732648 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3303
I20260620 14:56:51.732810 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967200138920-22143-0/minicluster-data/master-2/wal/instance
I20260620 14:56:51.733377 22143 external_mini_cluster.cc:1428] Setting key 2054ae56dfc4be4b71d5b6bee5ff93ef
I20260620 14:56:51.734293 3370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
I20260620 14:56:51.755364 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.737872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.21.159.254:36141)
0620 14:56:51.738097 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.738105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.738951 (+ 846us) server_negotiation.cc:407] Connection header received
0620 14:56:51.739479 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.739483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.739578 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.739664 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.741117 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.741893 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.743892 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.744581 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.744758 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.748760 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.748765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.748768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.748805 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.751959 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.752839 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.752847 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.752855 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.752981 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.753444 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.753448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.753450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.753724 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.753884 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.754728 (+ 844us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.754968 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:56:51.756675 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.735578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.21.159.252:46171)
0620 14:56:51.736393 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.736409 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.736470 (+ 61us) server_negotiation.cc:407] Connection header received
0620 14:56:51.739206 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.739233 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.739556 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.739767 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.741306 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.742835 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.744731 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.745661 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.745763 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.750134 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.750143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.750158 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.750189 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.753035 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.753793 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.753799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.753801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.753938 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.754357 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.754364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.754372 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.754719 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.755084 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.755406 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.755654 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":338,"threads_started":1}
I20260620 14:56:51.756729 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.737807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:36141 (local address 127.0.0.1:48698)
0620 14:56:51.738774 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.738828 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.739228 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.739769 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.739777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.740543 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.740931 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.740949 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.742054 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.742060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.743748 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.743759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.744887 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.744894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.745185 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.746056 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.746078 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.748541 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.752111 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.752125 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.752139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.752695 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.753108 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.753113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.753117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.753316 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.753882 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.753902 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.754285 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.754606 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.755188 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"spinlock_wait_cycles":15744,"thread_start_us":434,"threads_started":1}
I20260620 14:56:51.758440 3370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:51.773939 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.759134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:37749 (local address 127.0.0.1:48962)
0620 14:56:51.759382 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.759400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.759510 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.759884 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.759889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.760352 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.760555 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.760568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.762020 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.762024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.763440 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.763454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.764420 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.764424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.764576 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.765214 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.765237 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.767728 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.771242 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.771247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.771252 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.771798 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.772225 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.772231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.772234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.772393 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.772915 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.772920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.773085 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.773244 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.773544 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 14:56:51.773976 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.759256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.21.159.253:37749)
0620 14:56:51.759427 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.759431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.759447 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:56:51.759630 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.759637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.759719 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.759835 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.760707 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.761868 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.763589 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.764276 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.764408 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.767942 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.767947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.767950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.767983 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.771029 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.771932 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.771938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.771942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.772084 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.772512 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.772521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.772524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.772789 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.772939 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.773370 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.773588 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:56:51.775238 3370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:56:51.777734 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.760225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.21.159.254:36141)
0620 14:56:51.760481 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.760490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.760512 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:56:51.760573 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.760578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.760657 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.760745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.762312 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.763443 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.765227 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.765885 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.766030 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.770772 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.770777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.770781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.770816 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.773649 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.775073 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.775080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.775083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.775280 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.775710 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.775718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.775722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.776081 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.776403 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.777262 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.777462 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:56:51.791817 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.776167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48120)
0620 14:56:51.776659 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.776678 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.776807 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.777220 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.777223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.777689 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.777865 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.777876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.778849 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.778870 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.780062 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.780079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.780961 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.780969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.781175 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.782700 (+ 1525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.782725 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.785304 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.789064 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.789070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.789076 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.789441 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.789812 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.789820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.789825 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.790015 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.790592 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.790597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.790766 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.790928 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.791319 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412}
I20260620 14:56:51.791826 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.776390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.21.159.252:46171)
0620 14:56:51.776560 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.776566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.776730 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:56:51.776924 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.776929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.776996 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.777062 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.777992 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.778677 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.780227 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.780796 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.780905 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.785508 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.785514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.785517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.785546 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.788907 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.789551 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.789555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.789558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.789690 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.790128 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.790131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.790133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.790455 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.790620 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.791016 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.791485 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":52}
I20260620 14:56:51.796586 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.779710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.21.159.253:37749)
0620 14:56:51.779887 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.779892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.779910 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:56:51.780077 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.780081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.780169 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.780287 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.781729 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.782434 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.784334 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.784825 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.784966 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.790575 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.790581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.790584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.790616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.793747 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.794474 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.794479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.794483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.794642 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.795022 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.795027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.795030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.795423 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.795679 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.796005 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.796237 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:56:51.798255 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40: Bootstrap starting.
I20260620 14:56:51.801838 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:56:51.802214 3370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:51.803515 3370 log.cc:824] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:51.806933 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40: No bootstrap required, opened a new log
I20260620 14:56:51.817303 3221 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 4
I20260620 14:56:51.820551 3370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:51.821089 3370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:51.821223 3370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9920cc03284930bc32f55f26a7fd40, State: Initialized, Role: FOLLOWER
I20260620 14:56:51.822052 3370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:51.832286 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.807702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.21.159.254:36141)
0620 14:56:51.807898 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.807904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.808481 (+ 577us) server_negotiation.cc:407] Connection header received
0620 14:56:51.808766 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.808772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.808863 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.808935 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.811451 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.812595 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.814910 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.815709 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.815807 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.821604 (+ 5797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.821612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.821617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.821667 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.826969 (+ 5302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.828601 (+ 1632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.828611 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.828618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.828852 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.829496 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.829505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.829512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.830067 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.830430 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.831330 (+ 900us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.831694 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:56:51.835995 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.809437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.21.159.253:37749)
0620 14:56:51.809672 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.809682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.816013 (+ 6331us) server_negotiation.cc:407] Connection header received
0620 14:56:51.816220 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.816228 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.816307 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.816415 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.820304 (+ 3889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.821351 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.823575 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.824229 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.824325 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.828251 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.828258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.828262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.828306 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.832176 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.833167 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.833173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.833176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.833544 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.834035 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.834041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.834044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.834430 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.834663 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.835352 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.835621 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:56:51.837364 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:51.837882 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:56:51.839178 3370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:56:51.839704 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.807728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.21.159.252:46171)
0620 14:56:51.808099 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.808108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.815710 (+ 7602us) server_negotiation.cc:407] Connection header received
0620 14:56:51.816002 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.816007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.816094 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.816188 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.819795 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.821036 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.823260 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.824018 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.824144 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.830151 (+ 6007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.830161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.830494 (+ 333us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.830548 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.834345 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.835308 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.835314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.835317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.835511 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.837815 (+ 2304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.837821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.837824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.838239 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.838442 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.838951 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.839333 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967410, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
I20260620 14:56:51.853986 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.817879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48138)
0620 14:56:51.818052 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.818073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.818173 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.828271 (+ 10098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.828276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.828759 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.829374 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.829391 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.839420 (+ 10029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.839427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.840749 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.840767 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.841952 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.841958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.842147 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.842904 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.842923 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.845732 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.851026 (+ 5294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.851032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.851037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.851527 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.852015 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.852020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.852024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.852178 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.852848 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.852853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.853023 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.853188 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.853528 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:56:51.854048 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.818316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.21.159.252:46171)
0620 14:56:51.826133 (+ 7817us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.826139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.826165 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:56:51.826218 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.826222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.826312 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.836003 (+ 9691us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.836338 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.837631 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.840964 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.841767 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.841942 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.846007 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.846019 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.846027 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.846074 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.850801 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.851677 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.851683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.851687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.851872 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.852321 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.852326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.852331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.852676 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.852852 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.853329 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.853556 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7633,"thread_start_us":388,"threads_started":1}
I20260620 14:56:51.861806 3221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: Bootstrap starting.
I20260620 14:56:51.865712 3384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:56:51.866170 3384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:51.866847 3221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: Neither blocks nor log segments found. Creating new log.
W20260620 14:56:51.867333 3395 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:56:51.867461 3395 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:56:51.868211 3384 leader_election.cc:288] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02b1fa0b6804afcbf1a1f8547e05a70 (127.21.159.254:36141), 23e95847e2be48d8b30a6936bd962612 (127.21.159.253:37749)
I20260620 14:56:51.868364 3221 log.cc:824] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:51.850142 3303 master_runner.cc:186] Error getting master registration for 127.21.159.254:36141: OK, instance_id { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" instance_seqno: 1781967410665592 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:56:51.869576 3303 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:56:51.873951 3221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: No bootstrap required, opened a new log
I20260620 14:56:51.886423 3221 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:51.886874 3221 raft_consensus.cc:374] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:51.887003 3221 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02b1fa0b6804afcbf1a1f8547e05a70, State: Initialized, Role: FOLLOWER
I20260620 14:56:51.887892 3221 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:51.888954 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:51.889402 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:56:51.890537 3221 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:56:51.890578 3196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b9920cc03284930bc32f55f26a7fd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" is_pre_election: true
I20260620 14:56:51.891418 3196 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9920cc03284930bc32f55f26a7fd40 in term 0.
I20260620 14:56:51.892406 3321 leader_election.cc:302] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9920cc03284930bc32f55f26a7fd40, f02b1fa0b6804afcbf1a1f8547e05a70; no voters:
I20260620 14:56:51.895541 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:56:51.895792 3384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:56:51.895923 3384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:51.901315 3384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
W20260620 14:56:51.903167 3175 tablet.cc:2369] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:56:51.903509 3196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b9920cc03284930bc32f55f26a7fd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70"
I20260620 14:56:51.903965 3196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:51.908865 3384 leader_election.cc:288] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [CANDIDATE]: Term 1 election: Requested vote from peers f02b1fa0b6804afcbf1a1f8547e05a70 (127.21.159.254:36141), 23e95847e2be48d8b30a6936bd962612 (127.21.159.253:37749)
I20260620 14:56:51.910699 3196 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9920cc03284930bc32f55f26a7fd40 in term 1.
I20260620 14:56:51.911366 3321 leader_election.cc:302] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9920cc03284930bc32f55f26a7fd40, f02b1fa0b6804afcbf1a1f8547e05a70; no voters:
I20260620 14:56:51.911752 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:56:51.912528 3384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [term 1 LEADER]: Becoming Leader. State: Replica: 9b9920cc03284930bc32f55f26a7fd40, State: Running, Role: LEADER
I20260620 14:56:51.913386 3384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:51.919188 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9920cc03284930bc32f55f26a7fd40. Latest consensus state: current_term: 1 leader_uuid: "9b9920cc03284930bc32f55f26a7fd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:51.919946 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:51.924081 3403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:56:51.926467 3403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:56:51.928822 3295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } attempt: 5
Jun 20 14:56:51 dist-test-slave-3mqx krb5kdc[3141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967411, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
W20260620 14:56:51.954188 3324 tablet.cc:2369] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:56:51.957304 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.929462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:46171 (local address 127.0.0.1:48140)
0620 14:56:51.929840 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:51.929862 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:51.929982 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:51.930912 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:51.930918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:51.931481 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:51.931699 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.931711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.933308 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.933313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.934789 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:51.934804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.939572 (+ 4768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.939580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:51.939952 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.940921 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:51.940946 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:51.944089 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:51.948879 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.948885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.948890 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.949599 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.950275 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:51.950282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:51.950287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:51.953795 (+ 3508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:51.955190 (+ 1395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:51.955196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:51.955502 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 14:56:51.955748 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:56:51.956248 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260620 14:56:51.958863 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:51.930198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.21.159.252:46171)
0620 14:56:51.930397 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:51.930403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:51.930421 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:56:51.930479 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:51.930484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:51.930587 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:51.930692 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:51.931916 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.933062 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.938392 (+ 5330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:51.939174 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:51.939311 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:51.944423 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:51.944432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:51.944442 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:51.944484 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:51.948647 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.949824 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.949832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.949836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.950101 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:51.954068 (+ 3967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:51.954074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:51.954105 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:51.954920 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:51.956835 (+ 1915us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:51.957272 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 14:56:51.958256 (+ 984us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:56:51.964864 3295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612: Bootstrap starting.
I20260620 14:56:51.975889 3295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:51.981033 3295 log.cc:824] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:51.986348 3196 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [term 1 FOLLOWER]: Refusing update from remote peer 9b9920cc03284930bc32f55f26a7fd40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:56:51.987632 3401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:56:51.993656 3295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612: No bootstrap required, opened a new log
I20260620 14:56:52.014791 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9920cc03284930bc32f55f26a7fd40. Latest consensus state: current_term: 1 leader_uuid: "9b9920cc03284930bc32f55f26a7fd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:52.015193 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:56:52.015949 3295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:52.016395 3295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:52.016510 3295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23e95847e2be48d8b30a6936bd962612, State: Initialized, Role: FOLLOWER
I20260620 14:56:52.017490 3295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:52.023726 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b9920cc03284930bc32f55f26a7fd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:52.024192 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:52.026324 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b9920cc03284930bc32f55f26a7fd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:52.026672 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:56:52.027381 3413 mvcc.cc:205] Tried to move back new op lower bound from 7298938519474888704 to 7298938519205269504. Current Snapshot: MvccSnapshot[applied={T|T < 7298938519474888704}]
I20260620 14:56:52.029021 3403 catalog_manager.cc:1347] Generated new cluster ID: 83d615946b074bec86e6e5a292318dce
I20260620 14:56:52.029155 3403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:56:52.038741 3419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:52.039310 3419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:56:52.039564 3419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:56:52.039845 3419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } }
I20260620 14:56:52.041496 3419 leader_election.cc:288] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02b1fa0b6804afcbf1a1f8547e05a70 (127.21.159.254:36141), 9b9920cc03284930bc32f55f26a7fd40 (127.21.159.252:46171)
I20260620 14:56:52.041972 3196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23e95847e2be48d8b30a6936bd962612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" is_pre_election: true
I20260620 14:56:52.043852 3295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:56:52.044382 3268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:52.048771 3268 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 1 FOLLOWER]: Refusing update from remote peer 9b9920cc03284930bc32f55f26a7fd40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:56:52.044462 3270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b9920cc03284930bc32f55f26a7fd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23e95847e2be48d8b30a6936bd962612" is_pre_election: true
I20260620 14:56:52.048975 3269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b9920cc03284930bc32f55f26a7fd40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23e95847e2be48d8b30a6936bd962612"
I20260620 14:56:52.049348 3345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23e95847e2be48d8b30a6936bd962612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9920cc03284930bc32f55f26a7fd40" is_pre_election: true
I20260620 14:56:52.051286 3401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:56:52.040205 3154 master_runner.cc:186] Error getting master registration for 127.21.159.253:37749: OK, instance_id { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" instance_seqno: 1781967411194815 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:56:52.056095 3154 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:56:52.056248 3417 catalog_manager.cc:1259] Loaded cluster ID: 83d615946b074bec86e6e5a292318dce
I20260620 14:56:52.056366 3417 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: loading cluster ID for follower catalog manager: success
I20260620 14:56:52.057298 3247 leader_election.cc:302] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23e95847e2be48d8b30a6936bd962612; no voters: 9b9920cc03284930bc32f55f26a7fd40, f02b1fa0b6804afcbf1a1f8547e05a70
W20260620 14:56:52.059458 3417 catalog_manager.cc:1576] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 14:56:52.059849 3417 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:56:52.087339 3403 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:56:52.093735 3403 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:56:52.098367 3419 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:56:52.099421 3420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9920cc03284930bc32f55f26a7fd40. Latest consensus state: current_term: 1 leader_uuid: "9b9920cc03284930bc32f55f26a7fd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:52.099804 3420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:56:52.100178 3426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b9920cc03284930bc32f55f26a7fd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:52.100526 3426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:56:52.100381 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b9920cc03284930bc32f55f26a7fd40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02b1fa0b6804afcbf1a1f8547e05a70" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 36141 } } peers { permanent_uuid: "23e95847e2be48d8b30a6936bd962612" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 37749 } } peers { permanent_uuid: "9b9920cc03284930bc32f55f26a7fd40" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 46171 } } }
I20260620 14:56:52.100700 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40 [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:52.123103 3403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b9920cc03284930bc32f55f26a7fd40: Generated new TSK 0
I20260620 14:56:52.124294 3403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:56:52.153111 3436 catalog_manager.cc:1259] Loaded cluster ID: 83d615946b074bec86e6e5a292318dce
I20260620 14:56:52.153227 3436 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612: loading cluster ID for follower catalog manager: success
I20260620 14:56:52.157760 3436 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612: acquiring CA information for follower catalog manager: success
I20260620 14:56:52.160404 3436 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 23e95847e2be48d8b30a6936bd962612: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:56:53.063282 3417 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: acquiring CA information for follower catalog manager: success
I20260620 14:56:53.065303 3417 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f02b1fa0b6804afcbf1a1f8547e05a70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:56:53.251595 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3154
I20260620 14:56:53.253660 3216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:53.368781 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3154
I20260620 14:56:53.390542 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3228
I20260620 14:56:53.391731 3290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:53.506083 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3228
I20260620 14:56:53.526701 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3303
I20260620 14:56:53.528079 3365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:53.638384 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3303
2026-06-20T14:56:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4875 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:56:53 dist-test-slave-3mqx krb5kdc[3449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:56:53 dist-test-slave-3mqx krb5kdc[3449](info): set up 2 sockets
Jun 20 14:56:53 dist-test-slave-3mqx krb5kdc[3449](info): commencing operation
krb5kdc: starting...
W20260620 14:56:54.735550 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:56:54 dist-test-slave-3mqx krb5kdc[3449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:56:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:56:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:56:54.978883 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35635
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:35635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:55.317979 3462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:55.318302 3462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:55.318485 3462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:55.325961 3462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:56:55.326084 3462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:55.326171 3462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:55.326254 3462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:56:55.326316 3462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:56:55.336535 3462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:35635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:35635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:55.337945 3462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:55.339396 3462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:55.352748 3467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:55.352782 3468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:55.352841 3470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:55.353933 3462 server_base.cc:1034] running on GCE node
I20260620 14:56:55.354828 3462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:55.356477 3462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:55.357749 3462 hybrid_clock.cc:648] HybridClock initialized: now 1781967415357699 us; error 59 us; skew 500 ppm
I20260620 14:56:55.358150 3462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:55 dist-test-slave-3mqx krb5kdc[3449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967415, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:55.365947 3462 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:56:55.367619 3462 webserver.cc:460] Webserver started at http://127.21.159.254:40363/ using document root <none> and password file <none>
I20260620 14:56:55.368373 3462 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:55.368539 3462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:55.368896 3462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:55.372201 3462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "a27824855ffd4813a2a57419f05986ab"
format_stamp: "Formatted at 2026-06-20 14:56:55 on dist-test-slave-3mqx"
server_key: "e3d8d9f52ac339c3ccd96c2cf1883f34"
server_key_iv: "59f51d3eb2f1165433f5df5beb85f7db"
server_key_version: "encryptionkey@0"
I20260620 14:56:55.373014 3462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "a27824855ffd4813a2a57419f05986ab"
format_stamp: "Formatted at 2026-06-20 14:56:55 on dist-test-slave-3mqx"
server_key: "e3d8d9f52ac339c3ccd96c2cf1883f34"
server_key_iv: "59f51d3eb2f1165433f5df5beb85f7db"
server_key_version: "encryptionkey@0"
I20260620 14:56:55.378460 3462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:56:55.382284 3477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:55.383806 3462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:56:55.384001 3462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "a27824855ffd4813a2a57419f05986ab"
format_stamp: "Formatted at 2026-06-20 14:56:55 on dist-test-slave-3mqx"
server_key: "e3d8d9f52ac339c3ccd96c2cf1883f34"
server_key_iv: "59f51d3eb2f1165433f5df5beb85f7db"
server_key_version: "encryptionkey@0"
I20260620 14:56:55.384235 3462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:55.396471 3462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:55.401422 3462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:55.401791 3462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:55.421658 3462 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:35635
I20260620 14:56:55.421715 3528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:35635 every 8 connection(s)
I20260620 14:56:55.423252 3462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:56:55.427412 3529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:55.432345 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3462
I20260620 14:56:55.432595 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:56:55.433386 22143 external_mini_cluster.cc:1428] Setting key c9f2f3df00e913e9e6f34606dba2151e
I20260620 14:56:55.439383 3529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab: Bootstrap starting.
I20260620 14:56:55.442528 3529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:55.443928 3529 log.cc:824] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:55.447644 3529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab: No bootstrap required, opened a new log
Jun 20 14:56:55 dist-test-slave-3mqx krb5kdc[3449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:55.456293 3529 raft_consensus.cc:348] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27824855ffd4813a2a57419f05986ab" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35635 } }
I20260620 14:56:55.456715 3529 raft_consensus.cc:374] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:55.456843 3529 raft_consensus.cc:729] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27824855ffd4813a2a57419f05986ab, State: Initialized, Role: FOLLOWER
I20260620 14:56:55.457518 3529 consensus_queue.cc:260] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27824855ffd4813a2a57419f05986ab" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35635 } }
I20260620 14:56:55.457789 3529 raft_consensus.cc:388] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:56:55.457954 3529 raft_consensus.cc:482] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:56:55.458158 3529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:55.459918 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:55.435518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.21.159.254:35635)
0620 14:56:55.436447 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:55.436460 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:55.436508 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:56:55.438496 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:55.438524 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:55.438946 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:55.439463 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:55.441152 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:55.442576 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:55.444639 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:55.445811 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:55.445965 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:55.451219 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:55.451228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:55.451245 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:55.451286 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:55.454753 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:55.455700 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:55.455708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:55.455715 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:55.455899 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:55.456332 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:55.456336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:55.456338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:55.457072 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:55.457453 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:55.457990 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 14:56:55.458652 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":431,"threads_started":1}
I20260620 14:56:55.461031 3529 raft_consensus.cc:504] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27824855ffd4813a2a57419f05986ab" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35635 } }
I20260620 14:56:55.461658 3529 leader_election.cc:302] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27824855ffd4813a2a57419f05986ab; no voters:
I20260620 14:56:55.462313 3529 leader_election.cc:288] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:56:55.462457 3534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:56:55.464095 3534 raft_consensus.cc:686] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [term 1 LEADER]: Becoming Leader. State: Replica: a27824855ffd4813a2a57419f05986ab, State: Running, Role: LEADER
I20260620 14:56:55.464718 3529 sys_catalog.cc:564] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:56:55.464752 3534 consensus_queue.cc:237] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27824855ffd4813a2a57419f05986ab" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35635 } }
I20260620 14:56:55.464939 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:56:55.469019 3536 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader a27824855ffd4813a2a57419f05986ab. Latest consensus state: current_term: 1 leader_uuid: "a27824855ffd4813a2a57419f05986ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27824855ffd4813a2a57419f05986ab" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35635 } } }
I20260620 14:56:55.469413 3536 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:55.468602 3535 sys_catalog.cc:455] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a27824855ffd4813a2a57419f05986ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27824855ffd4813a2a57419f05986ab" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 35635 } } }
I20260620 14:56:55.470027 3535 sys_catalog.cc:458] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:55.470803 3541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:56:55.479020 3541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:56:55.490069 3541 catalog_manager.cc:1347] Generated new cluster ID: 935163b578e14e3ba8b4c99710102eb0
I20260620 14:56:55.490216 3541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:56:55.522183 3541 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:56:55.524489 3541 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:56:55.537103 3541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a27824855ffd4813a2a57419f05986ab: Generated new TSK 0
I20260620 14:56:55.538371 3541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:56:55.580233 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3462
I20260620 14:56:55.581700 3524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:55.677913 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3462
2026-06-20T14:56:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2035 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:56:55 dist-test-slave-3mqx krb5kdc[3557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:56:55 dist-test-slave-3mqx krb5kdc[3557](info): set up 2 sockets
Jun 20 14:56:55 dist-test-slave-3mqx krb5kdc[3557](info): commencing operation
krb5kdc: starting...
W20260620 14:56:56.796486 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:56:56 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:56:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:56:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:56:57.037052 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46849
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:46849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:57.368820 3570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:57.369120 3570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:57.369288 3570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:57.377694 3570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:56:57.377825 3570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:57.377874 3570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:57.377990 3570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:56:57.378050 3570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:56:57.388211 3570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:46849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:57.389639 3570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:57.391131 3570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:57.403785 3575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:57.403854 3576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:57.403851 3578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:57.404819 3570 server_base.cc:1034] running on GCE node
I20260620 14:56:57.405686 3570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:57.407296 3570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:57.408524 3570 hybrid_clock.cc:648] HybridClock initialized: now 1781967417408474 us; error 33 us; skew 500 ppm
I20260620 14:56:57.408950 3570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:57 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967417, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:57.412273 3570 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:56:57.413882 3570 webserver.cc:460] Webserver started at http://127.21.159.254:40343/ using document root <none> and password file <none>
I20260620 14:56:57.414553 3570 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:57.414654 3570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:57.414901 3570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:57.417958 3570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "2295bbb31d764a03a97fe219c171a1b2"
format_stamp: "Formatted at 2026-06-20 14:56:57 on dist-test-slave-3mqx"
server_key: "b09bacb0653293143f08c025c5df4f4d"
server_key_iv: "04891810bb7548eb664e01d7d286a66b"
server_key_version: "encryptionkey@0"
I20260620 14:56:57.418660 3570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "2295bbb31d764a03a97fe219c171a1b2"
format_stamp: "Formatted at 2026-06-20 14:56:57 on dist-test-slave-3mqx"
server_key: "b09bacb0653293143f08c025c5df4f4d"
server_key_iv: "04891810bb7548eb664e01d7d286a66b"
server_key_version: "encryptionkey@0"
I20260620 14:56:57.423939 3570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 14:56:57.427508 3585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:57.429039 3570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:56:57.429247 3570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "2295bbb31d764a03a97fe219c171a1b2"
format_stamp: "Formatted at 2026-06-20 14:56:57 on dist-test-slave-3mqx"
server_key: "b09bacb0653293143f08c025c5df4f4d"
server_key_iv: "04891810bb7548eb664e01d7d286a66b"
server_key_version: "encryptionkey@0"
I20260620 14:56:57.429441 3570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:57.471426 3570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:57.475943 3570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:57.476259 3570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:57.496294 3570 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:46849
I20260620 14:56:57.496318 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:46849 every 8 connection(s)
I20260620 14:56:57.497888 3570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:56:57.500643 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3570
I20260620 14:56:57.500862 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:56:57.501550 22143 external_mini_cluster.cc:1428] Setting key 9ab1869a4f18b93e1522ea0feff56567
I20260620 14:56:57.502177 3637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:57.515983 3637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Bootstrap starting.
Jun 20 14:56:57 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:57.519585 3637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:57.520788 3637 log.cc:824] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:57.524683 3637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: No bootstrap required, opened a new log
I20260620 14:56:57.526338 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:57.503839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.21.159.254:46849)
0620 14:56:57.504686 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:57.504698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:57.504740 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:56:57.507258 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:57.507289 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:57.507861 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:57.508302 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:57.509789 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:57.511205 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:57.512633 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:57.513374 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:57.513597 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:57.517845 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:57.517854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:57.517868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:57.517914 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:57.521129 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:57.521848 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:57.521855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:57.521861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:57.522050 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:57.522551 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:57.522559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:57.522567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:57.523364 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:57.523746 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:57.524270 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 14:56:57.524775 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":409,"threads_started":1}
I20260620 14:56:57.535260 3637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:56:57.535746 3637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:57.535866 3637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2295bbb31d764a03a97fe219c171a1b2, State: Initialized, Role: FOLLOWER
I20260620 14:56:57.536608 3637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:56:57.536878 3637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:56:57.537138 3637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:56:57.537331 3637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:57.540277 3637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:56:57.540885 3637 leader_election.cc:302] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2295bbb31d764a03a97fe219c171a1b2; no voters:
I20260620 14:56:57.541559 3637 leader_election.cc:288] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:56:57.541729 3642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:56:57.542574 3642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 LEADER]: Becoming Leader. State: Replica: 2295bbb31d764a03a97fe219c171a1b2, State: Running, Role: LEADER
I20260620 14:56:57.543277 3642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:56:57.544242 3637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:56:57.546514 3643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2295bbb31d764a03a97fe219c171a1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } } }
I20260620 14:56:57.546947 3643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:57.547492 3644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2295bbb31d764a03a97fe219c171a1b2. Latest consensus state: current_term: 1 leader_uuid: "2295bbb31d764a03a97fe219c171a1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } } }
I20260620 14:56:57.547813 3644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:56:57.549538 3651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:56:57.555032 3651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:56:57.564764 3651 catalog_manager.cc:1347] Generated new cluster ID: c59818c16900423c826d7ee68176147a
I20260620 14:56:57.564898 3651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:56:57.599819 3651 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:56:57.601958 3651 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:56:57.623304 3651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Generated new TSK 0
I20260620 14:56:57.624349 3651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:56:57.690739 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46849
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:58.029340 3664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:58.029645 3664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:58.029742 3664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:58.037561 3664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:58.037678 3664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:58.037801 3664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:56:58.048522 3664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:58.049975 3664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:58.051465 3664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:58.067579 3670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:58.067595 3669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:58.067530 3672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:58.069132 3664 server_base.cc:1034] running on GCE node
I20260620 14:56:58.069877 3664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:58.071100 3664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:58.072364 3664 hybrid_clock.cc:648] HybridClock initialized: now 1781967418072268 us; error 73 us; skew 500 ppm
I20260620 14:56:58.072748 3664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:58 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967418, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:58.076452 3664 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:56:58.078043 3664 webserver.cc:460] Webserver started at http://127.21.159.193:44799/ using document root <none> and password file <none>
I20260620 14:56:58.078773 3664 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:58.078933 3664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:58.079308 3664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:58.082259 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "02609d438bac4728afc0a9ad10cc5e3e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "9e3fc9f11d023e80abc37938e63f3205"
server_key_iv: "8d0702f299bff711098479eafd696ff3"
server_key_version: "encryptionkey@0"
I20260620 14:56:58.083242 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "02609d438bac4728afc0a9ad10cc5e3e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "9e3fc9f11d023e80abc37938e63f3205"
server_key_iv: "8d0702f299bff711098479eafd696ff3"
server_key_version: "encryptionkey@0"
I20260620 14:56:58.088762 3664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:56:58.092739 3679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:58.094272 3664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:56:58.094475 3664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "02609d438bac4728afc0a9ad10cc5e3e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "9e3fc9f11d023e80abc37938e63f3205"
server_key_iv: "8d0702f299bff711098479eafd696ff3"
server_key_version: "encryptionkey@0"
I20260620 14:56:58.094688 3664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:58.115247 3664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:58.119824 3664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:58.120126 3664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:58.121142 3664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:58.122591 3664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:58.122694 3664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:58.122869 3664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:58.122946 3664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:58.157653 3664 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:43027
I20260620 14:56:58.157701 3791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:43027 every 8 connection(s)
I20260620 14:56:58.158998 3664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:56:58.164491 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3664
I20260620 14:56:58.164758 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:56:58.165743 22143 external_mini_cluster.cc:1428] Setting key b415e3db372814aa81e95312cc15182f
Jun 20 14:56:58 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967418, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:58.185837 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:58.161995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:53183 (local address 127.21.159.254:46849)
0620 14:56:58.162468 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:58.162472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:58.163664 (+ 1192us) server_negotiation.cc:407] Connection header received
0620 14:56:58.166801 (+ 3137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:58.166805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:58.166886 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:58.166978 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:58.169327 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.170074 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.173014 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.173414 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.173520 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:58.178451 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:58.178460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:58.178468 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:58.178517 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:58.182233 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:58.183011 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:58.183019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:58.183025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:58.183203 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:58.183653 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:58.183661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:58.183667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:58.184076 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:58.184227 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:58.185070 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 14:56:58.185332 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":223,"threads_started":1}
I20260620 14:56:58.186509 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:58.162582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:53183)
0620 14:56:58.163475 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:58.163519 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:58.166431 (+ 2912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:58.167332 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:58.167342 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:58.168175 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:58.169104 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:58.169139 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.170264 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.170273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:58.172833 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:58.172856 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.173699 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.173711 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:58.174100 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:58.174960 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:58.175012 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:58.178160 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:58.182399 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:58.182407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:58.182420 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:58.182861 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:58.183315 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:58.183318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:58.183321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:58.183520 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:58.184253 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:58.184260 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:58.184607 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 14:56:58.184910 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 14:56:58.185335 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":306,"threads_started":1}
I20260620 14:56:58.188354 3792 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46849
I20260620 14:56:58.188674 3792 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:58.189442 3792 heartbeater.cc:511] Master 127.21.159.254:46849 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:56:58.191772 3602 ts_manager.cc:194] Registered new tserver with Master: 02609d438bac4728afc0a9ad10cc5e3e (127.21.159.193:43027)
I20260620 14:56:58.193347 3602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:53183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:56:58.220571 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46849
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:58.554064 3798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:58.554401 3798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:58.554579 3798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:58.562139 3798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:58.562321 3798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:58.562546 3798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:56:58.573320 3798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:58.574666 3798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:58.576141 3798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:58.590276 3803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:58.590317 3806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:58.590370 3804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:58.591272 3798 server_base.cc:1034] running on GCE node
I20260620 14:56:58.591959 3798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:58.592947 3798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:58.594203 3798 hybrid_clock.cc:648] HybridClock initialized: now 1781967418594138 us; error 57 us; skew 500 ppm
I20260620 14:56:58.594568 3798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:58 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967418, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:58.598534 3798 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:56:58.600178 3798 webserver.cc:460] Webserver started at http://127.21.159.194:41143/ using document root <none> and password file <none>
I20260620 14:56:58.600864 3798 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:58.600967 3798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:58.601217 3798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:58.604229 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "e64ac299b12d4ec4aac45337ef48172e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "a2ef4de17a71dbd565b4f0c7036089e7"
server_key_iv: "109c552217db03b2444561bf1654cf73"
server_key_version: "encryptionkey@0"
I20260620 14:56:58.604975 3798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "e64ac299b12d4ec4aac45337ef48172e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "a2ef4de17a71dbd565b4f0c7036089e7"
server_key_iv: "109c552217db03b2444561bf1654cf73"
server_key_version: "encryptionkey@0"
I20260620 14:56:58.610507 3798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:56:58.614121 3813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:58.615617 3798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:56:58.615746 3798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "e64ac299b12d4ec4aac45337ef48172e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "a2ef4de17a71dbd565b4f0c7036089e7"
server_key_iv: "109c552217db03b2444561bf1654cf73"
server_key_version: "encryptionkey@0"
I20260620 14:56:58.615911 3798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:58.635205 3798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:58.640026 3798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:58.640290 3798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:58.641252 3798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:58.642648 3798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:58.642721 3798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:58.642858 3798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:58.642925 3798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:58.678535 3798 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:42073
I20260620 14:56:58.678632 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:42073 every 8 connection(s)
I20260620 14:56:58.680013 3798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:56:58.684038 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3798
I20260620 14:56:58.684247 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:56:58.684794 22143 external_mini_cluster.cc:1428] Setting key 88c567cb505bf1ff4f9edaed294aa3cd
Jun 20 14:56:58 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967418, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:58.701495 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:58.682584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:48157 (local address 127.21.159.254:46849)
0620 14:56:58.682816 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:58.682822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:58.684371 (+ 1549us) server_negotiation.cc:407] Connection header received
0620 14:56:58.686953 (+ 2582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:58.686958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:58.687024 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:58.687194 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:58.688904 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.689614 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.691245 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.691660 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.691808 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:58.695658 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:58.695663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:58.695666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:58.695698 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:58.698273 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:58.699118 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:58.699128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:58.699130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:58.699262 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:58.699759 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:58.699770 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:58.699773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:58.700036 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:58.700200 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:58.700902 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 14:56:58.701137 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:56:58.702502 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:58.683189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:48157)
0620 14:56:58.684186 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:58.684242 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:58.686389 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:58.687477 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:58.687486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:58.688139 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:58.688727 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:58.688745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.689782 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.689787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:58.691097 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:58.691121 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:58.691895 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:58.691903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:58.692139 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:58.692912 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:58.692939 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:58.695418 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:58.698458 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:58.698467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:58.698507 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:58.698960 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:58.699385 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:58.699391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:58.699395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:58.699592 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:58.700169 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:58.700177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:58.700534 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 14:56:58.700821 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 14:56:58.701369 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":347,"threads_started":1}
I20260620 14:56:58.704063 3926 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46849
I20260620 14:56:58.704402 3926 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:58.705163 3926 heartbeater.cc:511] Master 127.21.159.254:46849 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
I20260620 14:56:58.706683 3602 ts_manager.cc:194] Registered new tserver with Master: e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073)
I20260620 14:56:58.707612 3602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:48157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:56:58.738019 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46849
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:56:59.081393 3931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:56:59.081717 3931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:56:59.081821 3931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:56:59.089359 3931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:56:59.089499 3931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:56:59.089625 3931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:56:59.100134 3931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:56:59.101732 3931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:56:59.103196 3931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:56:59.117494 3936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:59.117368 3939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:56:59.117456 3937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:56:59.118507 3931 server_base.cc:1034] running on GCE node
I20260620 14:56:59.119179 3931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:56:59.120321 3931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:56:59.121551 3931 hybrid_clock.cc:648] HybridClock initialized: now 1781967419121483 us; error 54 us; skew 500 ppm
I20260620 14:56:59.121892 3931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:56:59 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967419, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:56:59.125257 3931 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:56:59.126747 3931 webserver.cc:460] Webserver started at http://127.21.159.195:46705/ using document root <none> and password file <none>
I20260620 14:56:59.127518 3931 fs_manager.cc:359] Metadata directory not provided
I20260620 14:56:59.127630 3931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:56:59.127883 3931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:56:59.130842 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "ff83950e8e2240b3a49a643afb6b378c"
format_stamp: "Formatted at 2026-06-20 14:56:59 on dist-test-slave-3mqx"
server_key: "3329f2705c9c6748534bf84ddc0cd497"
server_key_iv: "f8974023498d45283d1252437c1a3d95"
server_key_version: "encryptionkey@0"
I20260620 14:56:59.131639 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "ff83950e8e2240b3a49a643afb6b378c"
format_stamp: "Formatted at 2026-06-20 14:56:59 on dist-test-slave-3mqx"
server_key: "3329f2705c9c6748534bf84ddc0cd497"
server_key_iv: "f8974023498d45283d1252437c1a3d95"
server_key_version: "encryptionkey@0"
I20260620 14:56:59.137161 3931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:56:59.140954 3946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:59.142412 3931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:56:59.142565 3931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "ff83950e8e2240b3a49a643afb6b378c"
format_stamp: "Formatted at 2026-06-20 14:56:59 on dist-test-slave-3mqx"
server_key: "3329f2705c9c6748534bf84ddc0cd497"
server_key_iv: "f8974023498d45283d1252437c1a3d95"
server_key_version: "encryptionkey@0"
I20260620 14:56:59.142768 3931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:56:59.173623 3931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:56:59.178483 3931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:56:59.178767 3931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:56:59.179837 3931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:56:59.181311 3931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:56:59.181396 3931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:59.181545 3931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:56:59.181640 3931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:56:59.196789 3792 heartbeater.cc:503] Master 127.21.159.254:46849 was elected leader, sending a full tablet report...
I20260620 14:56:59.220126 3931 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:33245
I20260620 14:56:59.220178 4058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:33245 every 8 connection(s)
I20260620 14:56:59.221630 3931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:56:59.222704 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 3931
I20260620 14:56:59.222913 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:56:59.223453 22143 external_mini_cluster.cc:1428] Setting key 1903d85a76b64d627961d267f626febd
Jun 20 14:56:59 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967419, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:56:59.243288 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.224648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:38389 (local address 127.21.159.254:46849)
0620 14:56:59.225115 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.225122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.226517 (+ 1395us) server_negotiation.cc:407] Connection header received
0620 14:56:59.228468 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.228472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.228544 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.228663 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:59.230120 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.230988 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.232714 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.233220 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.233322 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.236975 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:59.236981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:59.236984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:59.237017 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:59.240295 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.240981 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.240985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.240988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.241136 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.241536 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.241540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.241542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.241848 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:59.241962 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.242735 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.242914 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":222,"threads_started":1}
I20260620 14:56:59.244371 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.225465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:38389)
0620 14:56:59.226332 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:59.226386 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:59.228168 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:59.228845 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:59.228852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:59.229362 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:56:59.229943 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.229960 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.231210 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.231213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.232487 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.232499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.233415 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.233420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.233622 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.234385 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:56:59.234409 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:56:59.236757 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:56:59.240416 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:59.240423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:59.240437 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:59.240855 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:59.241241 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:56:59.241245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:56:59.241247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:56:59.241420 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:56:59.241995 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:56:59.242001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:56:59.242346 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 14:56:59.242587 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 14:56:59.243139 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":367,"threads_started":1}
I20260620 14:56:59.245962 4059 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46849
I20260620 14:56:59.246277 4059 heartbeater.cc:464] Registering TS with master...
I20260620 14:56:59.247044 4059 heartbeater.cc:511] Master 127.21.159.254:46849 requested a full tablet report, sending...
I20260620 14:56:59.248608 3602 ts_manager.cc:194] Registered new tserver with Master: ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245)
I20260620 14:56:59.249523 3602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:38389
I20260620 14:56:59.252494 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:56:59.270418 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.257561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.21.159.254:46849)
0620 14:56:59.257745 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.257750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.258250 (+ 500us) server_negotiation.cc:407] Connection header received
0620 14:56:59.258422 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.258426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.258494 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.258601 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:59.260109 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.260907 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.262193 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.262635 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.262741 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.265282 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:59.265287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:59.265290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:59.265322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:59.268065 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.268687 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.268691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.268694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.268820 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.269153 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.269156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.269158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.269507 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:59.269713 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.269985 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.270162 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:56:59.274425 3602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:56:59.281370 3602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:56:59.309312 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.295281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.21.159.195:33245)
0620 14:56:59.296320 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.296333 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.296877 (+ 544us) server_negotiation.cc:407] Connection header received
0620 14:56:59.297235 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.297247 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.297540 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.297752 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:59.299111 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.301397 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.305100 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.307140 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.307350 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.307593 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.308520 (+ 927us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.308794 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":690,"threads_started":1}
I20260620 14:56:59.311554 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.294515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:33245 (local address 127.0.0.1:49572)
0620 14:56:59.296781 (+ 2266us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:59.296798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:59.297010 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:59.298163 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:59.298174 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:59.298214 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:59.298809 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.298826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.301629 (+ 2803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.301636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.304872 (+ 3236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.304893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.307843 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.307853 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.308169 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.308245 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 14:56:59.309952 (+ 1707us) client_negotiation.cc:240] Negotiation successful
0620 14:56:59.310266 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2139,"thread_start_us":1995,"threads_started":1}
I20260620 14:56:59.313086 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.293886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:42073 (local address 127.0.0.1:60224)
0620 14:56:59.295693 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:59.295755 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:59.295970 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:59.298909 (+ 2939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:59.298917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:59.298952 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:59.299642 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.299659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.304191 (+ 4532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.304200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.308662 (+ 4462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.308692 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.312271 (+ 3579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.312278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.312457 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.312508 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:56:59.312595 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:56:59.312745 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1457,"thread_start_us":1278,"threads_started":1}
I20260620 14:56:59.313203 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.294016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.21.159.194:42073)
0620 14:56:59.296568 (+ 2552us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.296574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.296597 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:56:59.296688 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.296693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.296957 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.298605 (+ 1648us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:59.299902 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.303976 (+ 4074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.308925 (+ 4949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.312054 (+ 3129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.312244 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.312433 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.312670 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.312847 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2408,"thread_start_us":2301,"threads_started":1}
I20260620 14:56:59.315864 3994 tablet_service.cc:1467] Processing CreateTablet for tablet dcafa8e2461241a5a6a86c691ed54c4e (DEFAULT_TABLE table=test-table [id=4eee0a06feed4b85bd464b52004907de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:56:59.317466 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcafa8e2461241a5a6a86c691ed54c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:59.317051 3861 tablet_service.cc:1467] Processing CreateTablet for tablet dcafa8e2461241a5a6a86c691ed54c4e (DEFAULT_TABLE table=test-table [id=4eee0a06feed4b85bd464b52004907de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:56:59.318445 3861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcafa8e2461241a5a6a86c691ed54c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:59.319375 4076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.296841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43027 (local address 127.0.0.1:33266)
0620 14:56:59.310916 (+ 14075us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:59.310937 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:59.311081 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:59.312907 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:59.312912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:59.312939 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:59.313367 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.313378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.315238 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.315242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.316625 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.316642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.318478 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.318484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.318720 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.318744 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:56:59.318859 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:56:59.318987 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13937,"thread_start_us":14335,"threads_started":1}
I20260620 14:56:59.319432 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.297213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.21.159.193:43027)
0620 14:56:59.312138 (+ 14925us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.312147 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.312179 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:56:59.312281 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.312289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.312542 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.312761 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:59.313838 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.315011 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.316808 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.318228 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.318431 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.318603 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.318919 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.319100 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14732,"thread_start_us":14378,"threads_started":1}
I20260620 14:56:59.322608 3727 tablet_service.cc:1467] Processing CreateTablet for tablet dcafa8e2461241a5a6a86c691ed54c4e (DEFAULT_TABLE table=test-table [id=4eee0a06feed4b85bd464b52004907de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:56:59.324011 3727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcafa8e2461241a5a6a86c691ed54c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:56:59.333132 4079 tablet_bootstrap.cc:492] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Bootstrap starting.
I20260620 14:56:59.335548 4078 tablet_bootstrap.cc:492] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Bootstrap starting.
I20260620 14:56:59.337045 4080 tablet_bootstrap.cc:492] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Bootstrap starting.
I20260620 14:56:59.338853 4079 tablet_bootstrap.cc:654] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:59.339215 4078 tablet_bootstrap.cc:654] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:59.340039 4079 log.cc:824] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:59.340396 4080 tablet_bootstrap.cc:654] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Neither blocks nor log segments found. Creating new log.
I20260620 14:56:59.340452 4078 log.cc:824] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:59.341521 4080 log.cc:824] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:56:59.343411 4079 tablet_bootstrap.cc:492] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: No bootstrap required, opened a new log
I20260620 14:56:59.343453 4078 tablet_bootstrap.cc:492] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: No bootstrap required, opened a new log
I20260620 14:56:59.343731 4078 ts_tablet_manager.cc:1397] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260620 14:56:59.343713 4079 ts_tablet_manager.cc:1397] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260620 14:56:59.344520 4080 tablet_bootstrap.cc:492] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: No bootstrap required, opened a new log
I20260620 14:56:59.344820 4080 ts_tablet_manager.cc:1397] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 14:56:59.351455 4078 raft_consensus.cc:348] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.351867 4078 raft_consensus.cc:374] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:59.352082 4078 raft_consensus.cc:729] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64ac299b12d4ec4aac45337ef48172e, State: Initialized, Role: FOLLOWER
I20260620 14:56:59.352728 4078 consensus_queue.cc:260] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.353886 3926 heartbeater.cc:503] Master 127.21.159.254:46849 was elected leader, sending a full tablet report...
I20260620 14:56:59.354686 4078 ts_tablet_manager.cc:1428] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260620 14:56:59.354897 4079 raft_consensus.cc:348] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.355405 4079 raft_consensus.cc:374] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:59.355504 4079 raft_consensus.cc:729] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff83950e8e2240b3a49a643afb6b378c, State: Initialized, Role: FOLLOWER
I20260620 14:56:59.355299 4080 raft_consensus.cc:348] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.355720 4080 raft_consensus.cc:374] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:56:59.355818 4080 raft_consensus.cc:729] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02609d438bac4728afc0a9ad10cc5e3e, State: Initialized, Role: FOLLOWER
I20260620 14:56:59.356132 4079 consensus_queue.cc:260] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.356356 4080 consensus_queue.cc:260] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.358178 4079 ts_tablet_manager.cc:1428] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260620 14:56:59.358404 4059 heartbeater.cc:503] Master 127.21.159.254:46849 was elected leader, sending a full tablet report...
I20260620 14:56:59.360051 4080 ts_tablet_manager.cc:1428] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 14:56:59.411546 3793 tablet.cc:2369] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:56:59.432121 3927 tablet.cc:2369] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:56:59.473678 4060 tablet.cc:2369] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:56:59.673444 4086 raft_consensus.cc:482] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:56:59.673769 4086 raft_consensus.cc:504] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.675676 4086 leader_election.cc:288] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073), ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245)
I20260620 14:56:59.683794 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.676345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:47785 (local address 127.21.159.195:33245)
0620 14:56:59.676589 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.676599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.677313 (+ 714us) server_negotiation.cc:407] Connection header received
0620 14:56:59.677516 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.677523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.677588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.677695 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:59.678421 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.679703 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.682037 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.682815 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.683004 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.683097 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.683396 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.683523 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:56:59.683844 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.676239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:33245 (local address 127.21.159.193:47785)
0620 14:56:59.677206 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:59.677221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:59.677382 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:59.677711 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:59.677715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:59.677737 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:59.678200 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.678216 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.679842 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.679847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.681887 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.681897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.683020 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.683026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.683216 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.683241 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:56:59.683359 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:56:59.683492 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":607,"threads_started":1}
I20260620 14:56:59.684530 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.676081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:50333 (local address 127.21.159.194:42073)
0620 14:56:59.676330 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.676336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.677231 (+ 895us) server_negotiation.cc:407] Connection header received
0620 14:56:59.677480 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.677484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.677555 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.677704 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:56:59.678481 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.679322 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.681846 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.683323 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.683475 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.683560 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.684120 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.684262 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 14:56:59.685210 4087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.676189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:42073 (local address 127.21.159.193:50333)
0620 14:56:59.677097 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 14:56:59.677115 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:56:59.677314 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:56:59.677907 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:56:59.677912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:56:59.677930 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:56:59.678352 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.678364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.679464 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.679470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.681651 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:56:59.681666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.683724 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.683730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:56:59.683931 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.683969 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:56:59.684713 (+ 744us) client_negotiation.cc:240] Negotiation successful
0620 14:56:59.684875 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":487,"threads_started":1}
I20260620 14:56:59.685182 4014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcafa8e2461241a5a6a86c691ed54c4e" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff83950e8e2240b3a49a643afb6b378c" is_pre_election: true
I20260620 14:56:59.685801 4014 raft_consensus.cc:2454] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 0.
I20260620 14:56:59.686264 3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcafa8e2461241a5a6a86c691ed54c4e" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64ac299b12d4ec4aac45337ef48172e" is_pre_election: true
I20260620 14:56:59.686455 3683 leader_election.cc:302] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02609d438bac4728afc0a9ad10cc5e3e, ff83950e8e2240b3a49a643afb6b378c; no voters:
I20260620 14:56:59.686868 4086 raft_consensus.cc:2781] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:56:59.686831 3881 raft_consensus.cc:2454] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 0.
I20260620 14:56:59.687024 4086 raft_consensus.cc:482] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:56:59.687155 4086 raft_consensus.cc:3037] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:59.689538 4086 raft_consensus.cc:504] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.690225 4086 leader_election.cc:288] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 election: Requested vote from peers e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073), ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245)
I20260620 14:56:59.690675 3881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcafa8e2461241a5a6a86c691ed54c4e" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64ac299b12d4ec4aac45337ef48172e"
I20260620 14:56:59.690811 4014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcafa8e2461241a5a6a86c691ed54c4e" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff83950e8e2240b3a49a643afb6b378c"
I20260620 14:56:59.690907 3881 raft_consensus.cc:3037] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:59.691032 4014 raft_consensus.cc:3037] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:56:59.694324 4014 raft_consensus.cc:2454] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 1.
I20260620 14:56:59.694324 3881 raft_consensus.cc:2454] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 1.
I20260620 14:56:59.694839 3682 leader_election.cc:302] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02609d438bac4728afc0a9ad10cc5e3e, e64ac299b12d4ec4aac45337ef48172e; no voters:
I20260620 14:56:59.695245 4086 raft_consensus.cc:2781] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:56:59.695722 4086 raft_consensus.cc:686] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 1 LEADER]: Becoming Leader. State: Replica: 02609d438bac4728afc0a9ad10cc5e3e, State: Running, Role: LEADER
I20260620 14:56:59.696200 4086 consensus_queue.cc:237] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:56:59.701004 3601 catalog_manager.cc:5495] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 02609d438bac4728afc0a9ad10cc5e3e (127.21.159.193). New cstate: current_term: 1 leader_uuid: "02609d438bac4728afc0a9ad10cc5e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } health_report { overall_health: HEALTHY } } }
I20260620 14:56:59.765095 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.759373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.21.159.193:43027)
0620 14:56:59.759599 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.759607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.759689 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:56:59.759953 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.759957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.760036 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.760213 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:56:59.760607 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.761349 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.762779 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.763275 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.763407 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.763968 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:56:59.764493 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:56:59.764662 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.764744 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.764824 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:56:59.773315 3881 raft_consensus.cc:1261] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Refusing update from remote peer 02609d438bac4728afc0a9ad10cc5e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:56:59.773315 4014 raft_consensus.cc:1261] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Refusing update from remote peer 02609d438bac4728afc0a9ad10cc5e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:56:59.774327 4089 consensus_queue.cc:1035] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:56:59.774714 4086 consensus_queue.cc:1035] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:56:59.799887 3600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:56:59.800246 3600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:56:59.804422 3600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Sending DeleteTablet for 3 replicas of tablet dcafa8e2461241a5a6a86c691ed54c4e
I20260620 14:56:59.805652 3861 tablet_service.cc:1514] Processing DeleteTablet for tablet dcafa8e2461241a5a6a86c691ed54c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:56:59 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:60224
I20260620 14:56:59.805709 3994 tablet_service.cc:1514] Processing DeleteTablet for tablet dcafa8e2461241a5a6a86c691ed54c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:56:59 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:49572
I20260620 14:56:59.806430 3727 tablet_service.cc:1514] Processing DeleteTablet for tablet dcafa8e2461241a5a6a86c691ed54c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:56:59 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:33266
I20260620 14:56:59.806566 4102 tablet_replica.cc:330] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: stopping tablet replica
I20260620 14:56:59.806890 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3664
I20260620 14:56:59.807199 4102 raft_consensus.cc:2229] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:56:59.807837 4102 raft_consensus.cc:2258] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:56:59.810604 4102 ts_tablet_manager.cc:1905] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:56:59.817950 4104 tablet_replica.cc:330] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: stopping tablet replica
I20260620 14:56:59.818434 4104 raft_consensus.cc:2229] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:56:59.819195 4104 raft_consensus.cc:2258] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:56:59.819288 4102 ts_tablet_manager.cc:1918] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:56:59.819443 4102 log.cc:1196] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/dcafa8e2461241a5a6a86c691ed54c4e
I20260620 14:56:59.820114 4102 ts_tablet_manager.cc:1939] T dcafa8e2461241a5a6a86c691ed54c4e P e64ac299b12d4ec4aac45337ef48172e: Deleting consensus metadata
I20260620 14:56:59.820367 4103 tablet_replica.cc:330] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: stopping tablet replica
I20260620 14:56:59.820979 4103 raft_consensus.cc:2229] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:56:59.821492 4103 raft_consensus.cc:2258] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:56:59.822597 3588 catalog_manager.cc:4841] TS e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073): tablet dcafa8e2461241a5a6a86c691ed54c4e (table test-table [id=4eee0a06feed4b85bd464b52004907de]) successfully deleted
I20260620 14:56:59.823733 4104 ts_tablet_manager.cc:1905] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:56:59.824429 4103 ts_tablet_manager.cc:1905] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:56:59.831496 4104 ts_tablet_manager.cc:1918] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:56:59.831799 4104 log.cc:1196] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/dcafa8e2461241a5a6a86c691ed54c4e
I20260620 14:56:59.831951 4103 ts_tablet_manager.cc:1918] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:56:59.832144 4103 log.cc:1196] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/dcafa8e2461241a5a6a86c691ed54c4e
I20260620 14:56:59.832432 4104 ts_tablet_manager.cc:1939] T dcafa8e2461241a5a6a86c691ed54c4e P 02609d438bac4728afc0a9ad10cc5e3e: Deleting consensus metadata
I20260620 14:56:59.832693 4103 ts_tablet_manager.cc:1939] T dcafa8e2461241a5a6a86c691ed54c4e P ff83950e8e2240b3a49a643afb6b378c: Deleting consensus metadata
I20260620 14:56:59.834546 3589 catalog_manager.cc:4841] TS 02609d438bac4728afc0a9ad10cc5e3e (127.21.159.193:43027): tablet dcafa8e2461241a5a6a86c691ed54c4e (table test-table [id=4eee0a06feed4b85bd464b52004907de]) successfully deleted
I20260620 14:56:59.834882 3589 catalog_manager.cc:4841] TS ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245): tablet dcafa8e2461241a5a6a86c691ed54c4e (table test-table [id=4eee0a06feed4b85bd464b52004907de]) successfully deleted
Jun 20 14:56:59 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:56:59.846647 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.808150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.21.159.193:43027)
0620 14:56:59.808324 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.808330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.826775 (+ 18445us) server_negotiation.cc:407] Connection header received
0620 14:56:59.827307 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.827318 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.827415 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.827501 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:59.829763 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.831020 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.833648 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.834365 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.834478 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.839045 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:59.839114 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:59.839147 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:59.839186 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:59.842774 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.843837 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.843846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.843849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.844105 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.844581 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.844586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.844590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.845063 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:59.845462 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.845829 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.846121 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:56:59.847533 3787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:56:59.954259 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3664
I20260620 14:56:59.974309 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3798
Jun 20 14:56:59 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:56:59.990602 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:56:59.975193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.21.159.194:42073)
0620 14:56:59.975363 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:56:59.975368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:56:59.975467 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:56:59.975651 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:56:59.975655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:56:59.975719 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:56:59.975804 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:56:59.977167 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.977889 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.979424 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:56:59.979872 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:56:59.979991 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:56:59.984613 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:56:59.984628 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:56:59.984644 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:56:59.984689 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:56:59.987751 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.988471 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.988478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.988480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.988618 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:56:59.989026 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:56:59.989031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:56:59.989033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:56:59.989415 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:56:59.989734 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 14:56:59.990035 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:56:59.990264 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:56:59.991258 3921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:00.081630 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3798
I20260620 14:57:00.098701 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3931
Jun 20 14:57:00 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:57:00.114933 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:00.099652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.21.159.195:33245)
0620 14:57:00.099884 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:00.099890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:00.099980 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:57:00.100149 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:00.100153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:00.100223 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:00.100324 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:00.101598 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:00.102510 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:00.104045 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:00.104461 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:00.104622 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:00.109093 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:00.109110 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:00.109127 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:00.109166 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:00.112239 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:00.113013 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:00.113028 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:00.113032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:00.113218 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:00.113532 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:00.113535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:00.113537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:00.113910 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:00.114147 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:00.114444 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 14:57:00.114665 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:57:00.115727 4054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:00.205528 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3931
I20260620 14:57:00.222345 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 3570
I20260620 14:57:00.223737 3632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:00.337688 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 3570
I20260620 14:57:00.359615 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46849
--webserver_interface=127.21.159.254
--webserver_port=40343
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:46849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:00.697067 4110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:00.697372 4110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:00.697470 4110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:00.704753 4110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:00.704870 4110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:00.704918 4110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:00.705045 4110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:00.705109 4110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:00.715142 4110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:46849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=40343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:00.716547 4110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:00.717871 4110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:00.741364 4117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:00.741398 4116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:00.741323 4119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:00.742220 4110 server_base.cc:1034] running on GCE node
I20260620 14:57:00.743247 4110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:00.744915 4110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:00.746183 4110 hybrid_clock.cc:648] HybridClock initialized: now 1781967420746130 us; error 67 us; skew 500 ppm
I20260620 14:57:00.746629 4110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:00 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967420, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:00.757246 4110 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:57:00.759169 4110 webserver.cc:460] Webserver started at http://127.21.159.254:40343/ using document root <none> and password file <none>
I20260620 14:57:00.760005 4110 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:00.760182 4110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:00.766454 4110 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 14:57:00.769658 4126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:00.771030 4110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:57:00.771314 4110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "2295bbb31d764a03a97fe219c171a1b2"
format_stamp: "Formatted at 2026-06-20 14:56:57 on dist-test-slave-3mqx"
server_key: "b09bacb0653293143f08c025c5df4f4d"
server_key_iv: "04891810bb7548eb664e01d7d286a66b"
server_key_version: "encryptionkey@0"
I20260620 14:57:00.772408 4110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:00.817987 4110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:00.823114 4110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:00.823434 4110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:00.844043 4110 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:46849
I20260620 14:57:00.844070 4177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:46849 every 8 connection(s)
I20260620 14:57:00.845719 4110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:00.853171 4178 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:57:00.854808 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 4110
I20260620 14:57:00.855803 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:43027
--local_ip_for_outbound_sockets=127.21.159.193
--tserver_master_addrs=127.21.159.254:46849
--webserver_port=44799
--webserver_interface=127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:57:00.857103 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Bootstrap starting.
I20260620 14:57:00.878185 4178 log.cc:824] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:00.891251 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:57:00.891841 4178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Bootstrap complete.
I20260620 14:57:00.901252 4178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:00.902194 4178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2295bbb31d764a03a97fe219c171a1b2, State: Initialized, Role: FOLLOWER
I20260620 14:57:00.902878 4178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:00.903182 4178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:00.903324 4178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:00.903517 4178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:57:00.907981 4178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:00.908560 4178 leader_election.cc:302] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2295bbb31d764a03a97fe219c171a1b2; no voters:
I20260620 14:57:00.909233 4178 leader_election.cc:288] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:57:00.909379 4182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:57:00.910743 4182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 LEADER]: Becoming Leader. State: Replica: 2295bbb31d764a03a97fe219c171a1b2, State: Running, Role: LEADER
I20260620 14:57:00.911343 4182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:00.911787 4178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:00.913972 4183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2295bbb31d764a03a97fe219c171a1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } } }
I20260620 14:57:00.914431 4183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:00.914701 4184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2295bbb31d764a03a97fe219c171a1b2. Latest consensus state: current_term: 2 leader_uuid: "2295bbb31d764a03a97fe219c171a1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } } }
I20260620 14:57:00.915162 4184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:00.918092 4187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:00.926929 4187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:00.928610 4187 catalog_manager.cc:1259] Loaded cluster ID: c59818c16900423c826d7ee68176147a
I20260620 14:57:00.928800 4187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:00.932443 4187 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:57:00.950676 4187 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:57:00.960956 4187 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:00.962899 4187 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Loaded TSK: 0
I20260620 14:57:00.964509 4187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:57:01.216609 4180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:01.216944 4180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:01.217125 4180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:01.224924 4180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:01.225116 4180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:01.225299 4180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:57:01.235716 4180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:43027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=44799
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:01.237139 4180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:01.238641 4180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:01.258353 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:01.258401 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:01.259212 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:01.259889 4180 server_base.cc:1034] running on GCE node
I20260620 14:57:01.260762 4180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:01.262091 4180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:01.263355 4180 hybrid_clock.cc:648] HybridClock initialized: now 1781967421263274 us; error 68 us; skew 500 ppm
I20260620 14:57:01.263732 4180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:01 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:01.267198 4180 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:57:01.268978 4180 webserver.cc:460] Webserver started at http://127.21.159.193:44799/ using document root <none> and password file <none>
I20260620 14:57:01.269732 4180 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:01.269879 4180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:01.276044 4180 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:57:01.279628 4215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:01.281260 4180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:57:01.281461 4180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "02609d438bac4728afc0a9ad10cc5e3e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "9e3fc9f11d023e80abc37938e63f3205"
server_key_iv: "8d0702f299bff711098479eafd696ff3"
server_key_version: "encryptionkey@0"
I20260620 14:57:01.282567 4180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:01.317024 4180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:01.321897 4180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:01.322212 4180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:01.323341 4180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:01.324723 4180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:01.324822 4180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:01.324996 4180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:01.325093 4180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:01.359611 4180 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:43027
I20260620 14:57:01.359644 4327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:43027 every 8 connection(s)
I20260620 14:57:01.361267 4180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:01.370733 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 4180
I20260620 14:57:01.371629 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:42073
--local_ip_for_outbound_sockets=127.21.159.194
--tserver_master_addrs=127.21.159.254:46849
--webserver_port=41143
--webserver_interface=127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:57:01 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:01.386695 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:01.364430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:45535 (local address 127.21.159.254:46849)
0620 14:57:01.365182 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:01.365206 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:01.365717 (+ 511us) server_negotiation.cc:407] Connection header received
0620 14:57:01.367884 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:01.367898 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:01.368239 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:01.368574 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:01.370028 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.372139 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.373718 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.374856 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.375096 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:01.378792 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:01.378800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:01.378816 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:01.378859 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:01.381932 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:01.382735 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:01.382741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:01.382747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:01.382916 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:01.383392 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:01.383398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:01.383401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:01.383902 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:01.384115 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:01.384926 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 14:57:01.385325 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":279,"threads_started":1}
I20260620 14:57:01.386695 4331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:01.364773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:45535)
0620 14:57:01.365539 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:01.365578 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:01.367355 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:01.368700 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:01.368707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:01.369248 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:01.369794 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:01.369812 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.372324 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.372328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:01.373563 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:01.373575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.375139 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.375145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:01.375349 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:01.376129 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:01.376165 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:01.378487 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:01.382106 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:01.382116 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:01.382132 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:01.382576 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:01.383039 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:01.383045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:01.383049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:01.383273 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:01.384043 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:01.384052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:01.384405 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 14:57:01.384707 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 14:57:01.385285 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":314,"threads_started":1}
I20260620 14:57:01.389988 4328 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46849
I20260620 14:57:01.390332 4328 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:01.391193 4328 heartbeater.cc:511] Master 127.21.159.254:46849 requested a full tablet report, sending...
I20260620 14:57:01.393517 4143 ts_manager.cc:194] Registered new tserver with Master: 02609d438bac4728afc0a9ad10cc5e3e (127.21.159.193:43027)
I20260620 14:57:01.394989 4143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:45535
W20260620 14:57:01.718602 4332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:01.718902 4332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:01.718997 4332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:01.726737 4332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:01.726862 4332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:01.726979 4332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:57:01.737146 4332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:42073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=41143
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:01.738526 4332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:01.739980 4332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:01.754083 4337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:01.754305 4338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:01.754330 4340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:01.755875 4332 server_base.cc:1034] running on GCE node
I20260620 14:57:01.756588 4332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:01.757709 4332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:01.758949 4332 hybrid_clock.cc:648] HybridClock initialized: now 1781967421758890 us; error 54 us; skew 500 ppm
I20260620 14:57:01.759341 4332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:01 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:01.762633 4332 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:57:01.764248 4332 webserver.cc:460] Webserver started at http://127.21.159.194:41143/ using document root <none> and password file <none>
I20260620 14:57:01.764945 4332 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:01.765147 4332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:01.770643 4332 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:01.773461 4347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:01.774856 4332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:57:01.775110 4332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "e64ac299b12d4ec4aac45337ef48172e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "a2ef4de17a71dbd565b4f0c7036089e7"
server_key_iv: "109c552217db03b2444561bf1654cf73"
server_key_version: "encryptionkey@0"
I20260620 14:57:01.776292 4332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:01.829718 4332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:01.834558 4332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:01.834862 4332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:01.836005 4332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:01.837512 4332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:01.837636 4332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:01.837780 4332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:01.837872 4332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:01.878019 4332 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:42073
I20260620 14:57:01.878110 4459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:42073 every 8 connection(s)
I20260620 14:57:01.879597 4332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:57:01.888056 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 4332
I20260620 14:57:01.888804 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:33245
--local_ip_for_outbound_sockets=127.21.159.195
--tserver_master_addrs=127.21.159.254:46849
--webserver_port=46705
--webserver_interface=127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:57:01 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967421, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:01.902189 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:01.882370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:46821 (local address 127.21.159.254:46849)
0620 14:57:01.882586 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:01.882590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:01.883942 (+ 1352us) server_negotiation.cc:407] Connection header received
0620 14:57:01.885821 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:01.885826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:01.885916 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:01.886083 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:01.887862 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.889019 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.890659 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.891440 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.891597 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:01.895624 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:01.895630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:01.895634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:01.895675 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:01.898931 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:01.899813 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:01.899819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:01.899823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:01.899971 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:01.900418 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:01.900423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:01.900427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:01.900703 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:01.900868 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:01.901605 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 14:57:01.901841 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 14:57:01.903281 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:01.882855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:46821)
0620 14:57:01.883746 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:01.883791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:01.885531 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:01.886347 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:01.886356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:01.886907 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:01.887631 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:01.887653 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.889187 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.889191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:01.890488 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:01.890501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:01.891656 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:01.891662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:01.891864 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:01.892617 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:01.892640 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:01.895199 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:01.899183 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:01.899192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:01.899219 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:01.899677 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:01.900091 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:01.900096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:01.900100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:01.900292 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:01.900838 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:01.900845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:01.901203 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 14:57:01.901494 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 14:57:01.902057 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":413,"threads_started":1}
I20260620 14:57:01.904903 4460 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46849
I20260620 14:57:01.905220 4460 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:01.905959 4460 heartbeater.cc:511] Master 127.21.159.254:46849 requested a full tablet report, sending...
I20260620 14:57:01.907747 4143 ts_manager.cc:194] Registered new tserver with Master: e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073)
I20260620 14:57:01.908699 4143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:46821
W20260620 14:57:02.236936 4463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:02.237244 4463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:02.237341 4463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:02.244994 4463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:02.245126 4463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:02.245244 4463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:57:02.255396 4463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:33245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=46705
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:02.256757 4463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:02.258061 4463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:02.271548 4468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:02.271592 4471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:02.271504 4469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:02.272867 4463 server_base.cc:1034] running on GCE node
I20260620 14:57:02.273574 4463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:02.274613 4463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:02.275874 4463 hybrid_clock.cc:648] HybridClock initialized: now 1781967422275828 us; error 61 us; skew 500 ppm
I20260620 14:57:02.276216 4463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:02 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967422, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:02.279631 4463 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:57:02.281594 4463 webserver.cc:460] Webserver started at http://127.21.159.195:46705/ using document root <none> and password file <none>
I20260620 14:57:02.282339 4463 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:02.282510 4463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:02.288460 4463 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:57:02.291985 4478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:02.293644 4463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:57:02.293810 4463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "ff83950e8e2240b3a49a643afb6b378c"
format_stamp: "Formatted at 2026-06-20 14:56:59 on dist-test-slave-3mqx"
server_key: "3329f2705c9c6748534bf84ddc0cd497"
server_key_iv: "f8974023498d45283d1252437c1a3d95"
server_key_version: "encryptionkey@0"
I20260620 14:57:02.294838 4463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:02.313095 4463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:02.317610 4463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:02.317900 4463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:02.318966 4463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:02.320550 4463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:02.320636 4463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:02.320791 4463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:02.320875 4463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:02.358852 4590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:33245 every 8 connection(s)
I20260620 14:57:02.359622 4463 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:33245
I20260620 14:57:02.361940 4463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:02.364378 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 4463
Jun 20 14:57:02 dist-test-slave-3mqx krb5kdc[3557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967422, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:02.396699 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.367778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.21.159.254:46849)
0620 14:57:02.368007 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.368014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.374230 (+ 6216us) server_negotiation.cc:407] Connection header received
0620 14:57:02.374395 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.374401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.374494 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.374596 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:02.376780 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.379633 (+ 2853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.382007 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.383135 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.383245 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.387427 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:02.387433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:02.387437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:02.387485 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:02.392500 (+ 5015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:02.393805 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:02.393814 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:02.393819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:02.394012 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:02.394571 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:02.394578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:02.394583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:02.395023 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:02.395340 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.396032 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.396309 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":27392}
I20260620 14:57:02.398489 4328 heartbeater.cc:503] Master 127.21.159.254:46849 was elected leader, sending a full tablet report...
I20260620 14:57:02.399262 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.366866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:57983)
0620 14:57:02.372246 (+ 5380us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:02.372303 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:02.374206 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:02.375182 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:02.375202 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:02.376146 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:02.377132 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.377159 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.378709 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.378713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.380909 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.380927 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.381928 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.381935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.382302 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.383382 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:02.383442 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:02.386891 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:02.392878 (+ 5987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:02.392892 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:02.392932 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:02.393742 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:02.394282 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:02.394290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:02.394296 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:02.394611 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:02.395432 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:02.395443 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:02.396014 (+ 571us) client_negotiation.cc:769] Sending connection context
0620 14:57:02.396351 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 14:57:02.397370 (+ 1019us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5068,"thread_start_us":4816,"threads_started":1}
I20260620 14:57:02.401006 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.367778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:57983 (local address 127.21.159.254:46849)
0620 14:57:02.371627 (+ 3849us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.371637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.373606 (+ 1969us) server_negotiation.cc:407] Connection header received
0620 14:57:02.374642 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.374646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.374738 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.374845 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:02.377367 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.378517 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.381072 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.381668 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.381823 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.387357 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:02.387366 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:02.387374 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:02.387423 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:02.392628 (+ 5205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:02.393966 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:02.393974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:02.393978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:02.394126 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:02.394779 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:02.394786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:02.394791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:02.395258 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:02.400098 (+ 4840us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.400259 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.400520 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":343,"server-negotiator.queue_time_us":3679,"spinlock_wait_cycles":896,"thread_start_us":3285,"threads_started":1}
I20260620 14:57:02.402489 4591 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46849
I20260620 14:57:02.402909 4591 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:02.404081 4591 heartbeater.cc:511] Master 127.21.159.254:46849 requested a full tablet report, sending...
I20260620 14:57:02.405987 4142 ts_manager.cc:194] Registered new tserver with Master: ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245)
I20260620 14:57:02.407485 4142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:57983
I20260620 14:57:02.413565 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:57:02.420483 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.414470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.21.159.254:46849)
0620 14:57:02.414648 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.414653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.415214 (+ 561us) server_negotiation.cc:407] Connection header received
0620 14:57:02.415434 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.415439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.415590 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.415715 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:02.416126 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.416815 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.418343 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.418738 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.418831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.419373 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:02.419878 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:02.420006 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.420137 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.420205 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:57:02.421424 4142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:02.428905 4142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:02.466694 4602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.441427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:42073 (local address 127.0.0.1:60242)
0620 14:57:02.452869 (+ 11442us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:02.452923 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:02.453147 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:02.458458 (+ 5311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:02.458467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:02.458508 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:02.459036 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.459086 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.461340 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.461351 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.463504 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.463516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.465538 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.465544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.465754 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.465815 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 14:57:02.465948 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 14:57:02.466154 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11140,"mutex_wait_us":3,"thread_start_us":15476,"threads_started":1}
I20260620 14:57:02.467275 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.442029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.21.159.194:42073)
0620 14:57:02.457485 (+ 15456us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.457494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.457525 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:57:02.457626 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.457633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.457931 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.458384 (+ 453us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:02.459299 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.461164 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.463679 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.465309 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.465516 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.465720 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.466662 (+ 942us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.466876 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15267,"thread_start_us":16030,"threads_started":1}
I20260620 14:57:02.467504 4606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.456440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43027 (local address 127.0.0.1:33308)
0620 14:57:02.458613 (+ 2173us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:02.458632 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:02.458770 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:02.459706 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:02.459711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:02.459737 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:02.460125 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.460136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.462218 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.462225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.464402 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.464417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.466380 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.466389 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.466627 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.466674 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:57:02.466770 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:57:02.467140 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2008,"mutex_wait_us":4,"thread_start_us":1859,"threads_started":1}
I20260620 14:57:02.467871 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.441622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.21.159.195:33245)
0620 14:57:02.456149 (+ 14527us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.456164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.457416 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 14:57:02.458753 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.458761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.458986 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.459230 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:02.459747 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.461124 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.463824 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.465977 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.466173 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.466288 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.467177 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.467492 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13702,"spinlock_wait_cycles":531840,"thread_start_us":13766,"threads_started":1}
I20260620 14:57:02.468344 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.453872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.21.159.193:43027)
0620 14:57:02.458142 (+ 4270us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.458150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.458767 (+ 617us) server_negotiation.cc:407] Connection header received
0620 14:57:02.458961 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.458971 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.459244 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.459445 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:02.460377 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.462059 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.464583 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.465867 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.466057 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.466213 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.467831 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.468018 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4069,"thread_start_us":3865,"threads_started":1}
I20260620 14:57:02.469319 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.441597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:33245 (local address 127.0.0.1:49590)
0620 14:57:02.457249 (+ 15652us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:02.457277 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:02.458557 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:02.459218 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:02.459223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:02.459247 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:02.459582 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.459592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.461290 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.461294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.463658 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.463672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.466448 (+ 2776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.466453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.466735 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.466770 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:57:02.466864 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:57:02.466999 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15460,"mutex_wait_us":80,"thread_start_us":10779,"threads_started":1}
I20260620 14:57:02.471712 4263 tablet_service.cc:1467] Processing CreateTablet for tablet f832fc82e51242288a945e6c069f6d54 (DEFAULT_TABLE table=test-table [id=5dd92b5feb084143a1c2437a9223fb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:02.471585 4395 tablet_service.cc:1467] Processing CreateTablet for tablet f832fc82e51242288a945e6c069f6d54 (DEFAULT_TABLE table=test-table [id=5dd92b5feb084143a1c2437a9223fb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:02.472862 4263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f832fc82e51242288a945e6c069f6d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:02.473021 4395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f832fc82e51242288a945e6c069f6d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:02.473049 4526 tablet_service.cc:1467] Processing CreateTablet for tablet f832fc82e51242288a945e6c069f6d54 (DEFAULT_TABLE table=test-table [id=5dd92b5feb084143a1c2437a9223fb7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:02.474401 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f832fc82e51242288a945e6c069f6d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:02.485498 4607 tablet_bootstrap.cc:492] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Bootstrap starting.
I20260620 14:57:02.489672 4608 tablet_bootstrap.cc:492] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Bootstrap starting.
I20260620 14:57:02.490042 4607 tablet_bootstrap.cc:654] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:02.491437 4609 tablet_bootstrap.cc:492] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Bootstrap starting.
I20260620 14:57:02.491514 4607 log.cc:824] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:02.493283 4608 tablet_bootstrap.cc:654] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:02.494570 4608 log.cc:824] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:02.494853 4609 tablet_bootstrap.cc:654] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:02.495528 4607 tablet_bootstrap.cc:492] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: No bootstrap required, opened a new log
I20260620 14:57:02.496141 4609 log.cc:824] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:02.496382 4607 ts_tablet_manager.cc:1397] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260620 14:57:02.500595 4608 tablet_bootstrap.cc:492] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: No bootstrap required, opened a new log
I20260620 14:57:02.500896 4608 ts_tablet_manager.cc:1397] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 14:57:02.500944 4609 tablet_bootstrap.cc:492] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: No bootstrap required, opened a new log
I20260620 14:57:02.501187 4609 ts_tablet_manager.cc:1397] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 14:57:02.508508 4607 raft_consensus.cc:348] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.509029 4607 raft_consensus.cc:374] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:02.509128 4607 raft_consensus.cc:729] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02609d438bac4728afc0a9ad10cc5e3e, State: Initialized, Role: FOLLOWER
I20260620 14:57:02.509481 4608 raft_consensus.cc:348] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.509922 4608 raft_consensus.cc:374] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:02.509824 4607 consensus_queue.cc:260] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.510022 4608 raft_consensus.cc:729] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64ac299b12d4ec4aac45337ef48172e, State: Initialized, Role: FOLLOWER
I20260620 14:57:02.510659 4608 consensus_queue.cc:260] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.511875 4460 heartbeater.cc:503] Master 127.21.159.254:46849 was elected leader, sending a full tablet report...
I20260620 14:57:02.512869 4609 raft_consensus.cc:348] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.513506 4609 raft_consensus.cc:374] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:02.513604 4609 raft_consensus.cc:729] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff83950e8e2240b3a49a643afb6b378c, State: Initialized, Role: FOLLOWER
I20260620 14:57:02.514151 4608 ts_tablet_manager.cc:1428] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Time spent starting tablet: real 0.013s user 0.000s sys 0.013s
I20260620 14:57:02.514199 4609 consensus_queue.cc:260] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.515318 4591 heartbeater.cc:503] Master 127.21.159.254:46849 was elected leader, sending a full tablet report...
I20260620 14:57:02.516021 4609 ts_tablet_manager.cc:1428] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260620 14:57:02.517673 4607 ts_tablet_manager.cc:1428] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Time spent starting tablet: real 0.021s user 0.017s sys 0.000s
W20260620 14:57:02.613853 4329 tablet.cc:2369] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:02.620339 4592 tablet.cc:2369] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:02.631907 4461 tablet.cc:2369] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:57:02.725922 4613 raft_consensus.cc:482] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:02.726532 4613 raft_consensus.cc:504] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.728858 4613 leader_election.cc:288] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245), e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073)
I20260620 14:57:02.739320 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.729441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:52227 (local address 127.21.159.194:42073)
0620 14:57:02.729664 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.729672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.730721 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 14:57:02.730955 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.730960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.731040 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.731164 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:02.732208 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.733494 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.736793 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.738127 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.738297 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.738376 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.738873 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.739024 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 14:57:02.739320 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.729373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:55257 (local address 127.21.159.195:33245)
0620 14:57:02.729614 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.729621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.730825 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 14:57:02.730953 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.730958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.731040 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.731164 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:02.732208 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.733501 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.736800 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.738112 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.738297 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.738376 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.738893 (+ 517us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.739035 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:57:02.739986 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.729332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:33245 (local address 127.21.159.193:55257)
0620 14:57:02.730678 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:02.730700 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:02.730819 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:02.731247 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:02.731253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:02.731355 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:02.731980 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.731989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.733685 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.733691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.736555 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.736574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.738441 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.738446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.738634 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.738672 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:57:02.738790 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:57:02.738962 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"mutex_wait_us":205,"spinlock_wait_cycles":28416,"thread_start_us":653,"threads_started":1}
I20260620 14:57:02.740007 4617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.729296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:42073 (local address 127.21.159.193:52227)
0620 14:57:02.730558 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:02.730585 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:02.730790 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:02.731247 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:02.731253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:02.731349 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:02.731972 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.731987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.733690 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.733694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.736555 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:02.736574 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.738430 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.738437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:02.738649 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.738673 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:57:02.738786 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:57:02.738962 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"mutex_wait_us":302,"thread_start_us":563,"threads_started":1}
I20260620 14:57:02.741124 4415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832fc82e51242288a945e6c069f6d54" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64ac299b12d4ec4aac45337ef48172e" is_pre_election: true
I20260620 14:57:02.741122 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832fc82e51242288a945e6c069f6d54" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff83950e8e2240b3a49a643afb6b378c" is_pre_election: true
I20260620 14:57:02.741652 4415 raft_consensus.cc:2454] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 0.
I20260620 14:57:02.741652 4546 raft_consensus.cc:2454] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 0.
I20260620 14:57:02.742412 4219 leader_election.cc:302] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02609d438bac4728afc0a9ad10cc5e3e, e64ac299b12d4ec4aac45337ef48172e, ff83950e8e2240b3a49a643afb6b378c; no voters:
I20260620 14:57:02.742856 4613 raft_consensus.cc:2781] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:02.743000 4613 raft_consensus.cc:482] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:02.743157 4613 raft_consensus.cc:3037] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:02.745476 4613 raft_consensus.cc:504] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.746176 4613 leader_election.cc:288] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 election: Requested vote from peers ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245), e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073)
I20260620 14:57:02.746771 4415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832fc82e51242288a945e6c069f6d54" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64ac299b12d4ec4aac45337ef48172e"
I20260620 14:57:02.746781 4546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f832fc82e51242288a945e6c069f6d54" candidate_uuid: "02609d438bac4728afc0a9ad10cc5e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff83950e8e2240b3a49a643afb6b378c"
I20260620 14:57:02.746994 4415 raft_consensus.cc:3037] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:02.747010 4546 raft_consensus.cc:3037] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:02.749343 4546 raft_consensus.cc:2454] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 1.
I20260620 14:57:02.749382 4415 raft_consensus.cc:2454] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02609d438bac4728afc0a9ad10cc5e3e in term 1.
I20260620 14:57:02.749920 4219 leader_election.cc:302] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02609d438bac4728afc0a9ad10cc5e3e, ff83950e8e2240b3a49a643afb6b378c; no voters:
I20260620 14:57:02.750324 4613 raft_consensus.cc:2781] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:02.750811 4613 raft_consensus.cc:686] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 1 LEADER]: Becoming Leader. State: Replica: 02609d438bac4728afc0a9ad10cc5e3e, State: Running, Role: LEADER
I20260620 14:57:02.751530 4613 consensus_queue.cc:237] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } }
I20260620 14:57:02.756525 4142 catalog_manager.cc:5495] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 02609d438bac4728afc0a9ad10cc5e3e (127.21.159.193). New cstate: current_term: 1 leader_uuid: "02609d438bac4728afc0a9ad10cc5e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02609d438bac4728afc0a9ad10cc5e3e" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43027 } health_report { overall_health: HEALTHY } } }
I20260620 14:57:02.837294 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.831557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.21.159.193:43027)
0620 14:57:02.831821 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.831826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.831845 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:57:02.831939 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.831943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.832026 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.832162 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:02.832520 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.833336 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.834722 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.835265 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.835371 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.835956 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:02.836590 (+ 634us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:02.836731 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.836893 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.837010 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:57:02.846352 4415 raft_consensus.cc:1261] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Refusing update from remote peer 02609d438bac4728afc0a9ad10cc5e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:02.846385 4546 raft_consensus.cc:1261] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Refusing update from remote peer 02609d438bac4728afc0a9ad10cc5e3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:02.847383 4618 consensus_queue.cc:1035] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64ac299b12d4ec4aac45337ef48172e" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 42073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:02.847811 4613 consensus_queue.cc:1035] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff83950e8e2240b3a49a643afb6b378c" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:02.860119 4621 mvcc.cc:205] Tried to move back new op lower bound from 7298938563967176704 to 7298938563594846208. Current Snapshot: MvccSnapshot[applied={T|T < 7298938563967176704}]
I20260620 14:57:02.879900 4142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:02.880153 4142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:02.883667 4142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Sending DeleteTablet for 3 replicas of tablet f832fc82e51242288a945e6c069f6d54
I20260620 14:57:02.884791 4526 tablet_service.cc:1514] Processing DeleteTablet for tablet f832fc82e51242288a945e6c069f6d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:02 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:49590
I20260620 14:57:02.884792 4395 tablet_service.cc:1514] Processing DeleteTablet for tablet f832fc82e51242288a945e6c069f6d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:02 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:60242
I20260620 14:57:02.889511 4263 tablet_service.cc:1514] Processing DeleteTablet for tablet f832fc82e51242288a945e6c069f6d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:02 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:33308
I20260620 14:57:02.898116 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 4180
I20260620 14:57:02.910763 4632 tablet_replica.cc:330] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: stopping tablet replica
I20260620 14:57:02.911322 4632 raft_consensus.cc:2229] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:02.912053 4632 raft_consensus.cc:2258] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:02.917732 4632 ts_tablet_manager.cc:1905] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:02.919564 4630 tablet_replica.cc:330] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: stopping tablet replica
I20260620 14:57:02.920043 4630 raft_consensus.cc:2229] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:02.920562 4630 raft_consensus.cc:2258] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:02.921393 4631 tablet_replica.cc:330] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: stopping tablet replica
I20260620 14:57:02.921810 4631 raft_consensus.cc:2229] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:02.922302 4631 raft_consensus.cc:2258] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:02.924788 4631 ts_tablet_manager.cc:1905] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:02.929018 4630 ts_tablet_manager.cc:1905] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:02.931787 4631 ts_tablet_manager.cc:1918] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:02.931939 4631 log.cc:1196] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/f832fc82e51242288a945e6c069f6d54
I20260620 14:57:02.932467 4631 ts_tablet_manager.cc:1939] T f832fc82e51242288a945e6c069f6d54 P e64ac299b12d4ec4aac45337ef48172e: Deleting consensus metadata
I20260620 14:57:02.933674 4632 ts_tablet_manager.cc:1918] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:02.933842 4632 log.cc:1196] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/f832fc82e51242288a945e6c069f6d54
I20260620 14:57:02.934369 4632 ts_tablet_manager.cc:1939] T f832fc82e51242288a945e6c069f6d54 P 02609d438bac4728afc0a9ad10cc5e3e: Deleting consensus metadata
I20260620 14:57:02.934442 4129 catalog_manager.cc:4841] TS e64ac299b12d4ec4aac45337ef48172e (127.21.159.194:42073): tablet f832fc82e51242288a945e6c069f6d54 (table test-table [id=5dd92b5feb084143a1c2437a9223fb7f]) successfully deleted
I20260620 14:57:02.935205 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:02.899901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.21.159.193:43027)
0620 14:57:02.900123 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:02.900129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:02.907730 (+ 7601us) server_negotiation.cc:407] Connection header received
0620 14:57:02.908037 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:02.908044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:02.908152 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:02.908236 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:02.912431 (+ 4195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.913762 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.922764 (+ 9002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:02.923421 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:02.923582 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:02.926537 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:02.926550 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:02.926564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:02.926604 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:02.930592 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:02.931671 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:02.931681 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:02.931685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:02.931956 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:02.932521 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:02.932602 (+ 81us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:02.932606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:02.933040 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:02.933317 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:02.934486 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0620 14:57:02.934801 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:57:02.935899 4630 ts_tablet_manager.cc:1918] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:02.936081 4630 log.cc:1196] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/f832fc82e51242288a945e6c069f6d54
I20260620 14:57:02.936133 4130 catalog_manager.cc:4841] TS 02609d438bac4728afc0a9ad10cc5e3e (127.21.159.193:43027): tablet f832fc82e51242288a945e6c069f6d54 (table test-table [id=5dd92b5feb084143a1c2437a9223fb7f]) successfully deleted
I20260620 14:57:02.936596 4630 ts_tablet_manager.cc:1939] T f832fc82e51242288a945e6c069f6d54 P ff83950e8e2240b3a49a643afb6b378c: Deleting consensus metadata
I20260620 14:57:02.937422 4323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:02.938313 4130 catalog_manager.cc:4841] TS ff83950e8e2240b3a49a643afb6b378c (127.21.159.195:33245): tablet f832fc82e51242288a945e6c069f6d54 (table test-table [id=5dd92b5feb084143a1c2437a9223fb7f]) successfully deleted
I20260620 14:57:03.038681 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 4180
I20260620 14:57:03.058202 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 4332
I20260620 14:57:03.073262 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:03.059138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.21.159.194:42073)
0620 14:57:03.059299 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:03.059303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:03.059321 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:57:03.059381 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:03.059385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:03.059446 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:03.059530 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:03.060909 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:03.061582 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:03.063393 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:03.063782 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:03.063894 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:03.067299 (+ 3405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:03.067321 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:03.067341 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:03.067393 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:03.070335 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:03.071138 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:03.071146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:03.071148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:03.071311 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:03.071737 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:03.071740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:03.071742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:03.072126 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:03.072336 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:03.072804 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 14:57:03.072994 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:57:03.073978 4455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:03.167596 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 4332
I20260620 14:57:03.184912 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 4463
I20260620 14:57:03.200829 4603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:03.185881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.21.159.195:33245)
0620 14:57:03.186061 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:03.186065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:03.186205 (+ 140us) server_negotiation.cc:407] Connection header received
0620 14:57:03.186394 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:03.186399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:03.186480 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:03.186569 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:03.188132 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:03.188978 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:03.190617 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:03.191073 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:03.191245 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:03.193971 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:03.193986 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:03.194003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:03.194043 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:03.197116 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:03.198022 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:03.198051 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:03.198057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:03.198260 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:03.198780 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:03.198785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:03.198789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:03.199526 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:03.199870 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:03.200230 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:57:03.200465 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:57:03.201483 4586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:03.294070 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 4463
I20260620 14:57:03.310977 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 4110
I20260620 14:57:03.312459 4173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:03.437966 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 4110
I20260620 14:57:03.460508 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46849
--webserver_interface=127.21.159.254
--webserver_port=40343
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:46849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:03.803596 4638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:03.803907 4638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:03.804076 4638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:03.811498 4638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:03.811616 4638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:03.811728 4638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:03.811879 4638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:03.811967 4638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:03.821969 4638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:46849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=40343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:03.823509 4638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:03.824903 4638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:03.840090 4643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:03.840137 4646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:03.840101 4644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:03.840945 4638 server_base.cc:1034] running on GCE node
I20260620 14:57:03.841786 4638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:03.843487 4638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:03.844760 4638 hybrid_clock.cc:648] HybridClock initialized: now 1781967423844697 us; error 53 us; skew 500 ppm
I20260620 14:57:03.845201 4638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:03 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967423, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:03.848814 4638 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:57:03.850310 4638 webserver.cc:460] Webserver started at http://127.21.159.254:40343/ using document root <none> and password file <none>
I20260620 14:57:03.851028 4638 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:03.851212 4638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:03.857736 4638 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:57:03.863189 4653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:03.864815 4638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:03.864989 4638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "2295bbb31d764a03a97fe219c171a1b2"
format_stamp: "Formatted at 2026-06-20 14:56:57 on dist-test-slave-3mqx"
server_key: "b09bacb0653293143f08c025c5df4f4d"
server_key_iv: "04891810bb7548eb664e01d7d286a66b"
server_key_version: "encryptionkey@0"
I20260620 14:57:03.866104 4638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:03.881892 4638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:03.886534 4638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:03.886770 4638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:03.906472 4638 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:46849
I20260620 14:57:03.906499 4704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:46849 every 8 connection(s)
I20260620 14:57:03.908041 4638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:03.914726 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 4638
I20260620 14:57:03.915755 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:43027
--local_ip_for_outbound_sockets=127.21.159.193
--tserver_master_addrs=127.21.159.254:46849
--webserver_port=44799
--webserver_interface=127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:57:03.916102 4705 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:57:03.919814 4705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Bootstrap starting.
I20260620 14:57:03.942368 4705 log.cc:824] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:03.960443 4705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:57:03.961081 4705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2: Bootstrap complete.
I20260620 14:57:03.971824 4705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:03.972851 4705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2295bbb31d764a03a97fe219c171a1b2, State: Initialized, Role: FOLLOWER
I20260620 14:57:03.973592 4705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:03.973851 4705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:03.973978 4705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:03.974206 4705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:57:03.978466 4705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:03.979040 4705 leader_election.cc:302] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2295bbb31d764a03a97fe219c171a1b2; no voters:
I20260620 14:57:03.979863 4705 leader_election.cc:288] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:57:03.980015 4709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:57:03.981268 4709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [term 3 LEADER]: Becoming Leader. State: Replica: 2295bbb31d764a03a97fe219c171a1b2, State: Running, Role: LEADER
I20260620 14:57:03.981968 4709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } }
I20260620 14:57:03.982342 4705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:03.984930 4711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2295bbb31d764a03a97fe219c171a1b2. Latest consensus state: current_term: 3 leader_uuid: "2295bbb31d764a03a97fe219c171a1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } } }
I20260620 14:57:03.985400 4711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:03.986164 4710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2295bbb31d764a03a97fe219c171a1b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295bbb31d764a03a97fe219c171a1b2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46849 } } }
I20260620 14:57:03.986589 4710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295bbb31d764a03a97fe219c171a1b2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:03.995923 4714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:04.008368 4714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:04.010620 4714 catalog_manager.cc:1259] Loaded cluster ID: c59818c16900423c826d7ee68176147a
I20260620 14:57:04.011155 4714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:57:04.013526 4714 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1a37aed9ad google::LogMessage::Fail() at ??:0
@ 0x7f1a37af088a google::LogMessage::SendToLog() at ??:0
@ 0x7f1a37aed4de google::LogMessage::Flush() at ??:0
@ 0x7f1a37aeea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1a4d520512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1a4d51d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1a4d5a5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1a4d5a5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1a3a739345 kudu::ThreadPool::DispatchThread() at ??:0
I20260620 14:57:04.076530 4732 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f1a3a73c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1a3a73c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1a3a717eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1a3d9f16db start_thread at ??:0
@ 0x7f1a3698761f clone at ??:0
W20260620 14:57:04.346016 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:04.346356 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:04.346526 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:04.354537 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:04.354662 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:04.354842 4707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:57:04.365276 4707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:43027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=44799
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:04.366721 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:04.368151 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:04.381793 4747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:04.381878 4746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:04.381902 4749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:04.382660 4707 server_base.cc:1034] running on GCE node
I20260620 14:57:04.383424 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:04.384413 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:04.385663 4707 hybrid_clock.cc:648] HybridClock initialized: now 1781967424385613 us; error 43 us; skew 500 ppm
I20260620 14:57:04.386014 4707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:04 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967424, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:04.389302 4707 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:57:04.390746 4707 webserver.cc:460] Webserver started at http://127.21.159.193:44799/ using document root <none> and password file <none>
I20260620 14:57:04.391474 4707 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:04.391584 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:04.398475 4707 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:57:04.401408 4756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:04.402916 4707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:04.403141 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "02609d438bac4728afc0a9ad10cc5e3e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "9e3fc9f11d023e80abc37938e63f3205"
server_key_iv: "8d0702f299bff711098479eafd696ff3"
server_key_version: "encryptionkey@0"
I20260620 14:57:04.404209 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:04.432423 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:04.436911 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:04.437244 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:04.438336 4707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:04.439831 4707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:04.439958 4707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:04.440096 4707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:04.440186 4707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:04.476635 4707 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:43027
I20260620 14:57:04.476732 4868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:43027 every 8 connection(s)
I20260620 14:57:04.478194 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:04.482736 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 4707
I20260620 14:57:04.483561 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:42073
--local_ip_for_outbound_sockets=127.21.159.194
--tserver_master_addrs=127.21.159.254:46849
--webserver_port=41143
--webserver_interface=127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:57:04.483781 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:04.481316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:47243)
0620 14:57:04.482105 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:04.482499 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":303,"threads_started":1}
W20260620 14:57:04.484537 4869 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:46849 (0 consecutive failures): Network error: Failed to ping master at 127.21.159.254:46849: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
I20260620 14:57:04.485541 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:04.485087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:53903)
0620 14:57:04.485290 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:04.485384 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:57:04.486575 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:04.486115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:43249)
0620 14:57:04.486312 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:04.486416 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 14:57:04.486858 4869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:57:04.819175 4872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:04.819465 4872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:04.819638 4872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:04.826928 4872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:04.827044 4872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:04.827299 4872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:57:04.838480 4872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:42073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=41143
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:04.839901 4872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:04.841248 4872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:04.854758 4877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:04.855098 4878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:04.855602 4880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:04.856202 4872 server_base.cc:1034] running on GCE node
I20260620 14:57:04.856788 4872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:04.857733 4872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:04.858947 4872 hybrid_clock.cc:648] HybridClock initialized: now 1781967424858905 us; error 31 us; skew 500 ppm
I20260620 14:57:04.859375 4872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:04 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967424, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:04.862738 4872 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:57:04.864398 4872 webserver.cc:460] Webserver started at http://127.21.159.194:41143/ using document root <none> and password file <none>
I20260620 14:57:04.865067 4872 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:04.865182 4872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:04.870621 4872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 14:57:04.873415 4887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:04.874941 4872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:57:04.875155 4872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "e64ac299b12d4ec4aac45337ef48172e"
format_stamp: "Formatted at 2026-06-20 14:56:58 on dist-test-slave-3mqx"
server_key: "a2ef4de17a71dbd565b4f0c7036089e7"
server_key_iv: "109c552217db03b2444561bf1654cf73"
server_key_version: "encryptionkey@0"
I20260620 14:57:04.876183 4872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:04.892980 4872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:04.897226 4872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:04.897521 4872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:04.898545 4872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:04.899956 4872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:04.900043 4872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:04.900199 4872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:04.900307 4872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:04.937271 4872 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:42073
I20260620 14:57:04.937306 4999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:42073 every 8 connection(s)
I20260620 14:57:04.938869 4872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:57:04.944201 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:04.942143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:37569)
0620 14:57:04.942940 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:04.943264 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":294,"threads_started":1}
W20260620 14:57:04.944913 5000 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:46849 (0 consecutive failures): Network error: Failed to ping master at 127.21.159.254:46849: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
I20260620 14:57:04.945755 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:04.945417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:38931)
0620 14:57:04.945571 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:04.945647 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 14:57:04.946619 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:04.946277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:54979)
0620 14:57:04.946423 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:04.946511 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:57:04.946859 5000 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:57:04.948230 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 4872
I20260620 14:57:04.948980 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:33245
--local_ip_for_outbound_sockets=127.21.159.195
--tserver_master_addrs=127.21.159.254:46849
--webserver_port=46705
--webserver_interface=127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:05.280042 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:05.280349 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:05.280572 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:05.287972 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:05.288091 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:05.288303 5003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:57:05.298249 5003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:33245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=46705
--tserver_master_addrs=127.21.159.254:46849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:05.299638 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:05.301005 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:05.314901 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:05.314846 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:05.314880 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:05.316998 5003 server_base.cc:1034] running on GCE node
I20260620 14:57:05.317773 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:05.318902 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:05.320178 5003 hybrid_clock.cc:648] HybridClock initialized: now 1781967425320156 us; error 97 us; skew 500 ppm
I20260620 14:57:05.320574 5003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:05 dist-test-slave-3mqx krb5kdc[3557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967425, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:05.324244 5003 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:57:05.325955 5003 webserver.cc:460] Webserver started at http://127.21.159.195:46705/ using document root <none> and password file <none>
I20260620 14:57:05.326668 5003 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:05.326771 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:05.332526 5003 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:57:05.335501 5018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:05.337031 5003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:57:05.337244 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "ff83950e8e2240b3a49a643afb6b378c"
format_stamp: "Formatted at 2026-06-20 14:56:59 on dist-test-slave-3mqx"
server_key: "3329f2705c9c6748534bf84ddc0cd497"
server_key_iv: "f8974023498d45283d1252437c1a3d95"
server_key_version: "encryptionkey@0"
I20260620 14:57:05.338285 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:05.366474 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:05.371030 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:05.371485 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:05.372653 5003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:05.374101 5003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:05.374230 5003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:05.374366 5003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:05.374454 5003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:05.410696 5003 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:33245
I20260620 14:57:05.410791 5130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:33245 every 8 connection(s)
I20260620 14:57:05.412245 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:05.413259 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5003
I20260620 14:57:05.417707 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:05.415682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:56889)
0620 14:57:05.416566 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:05.416816 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":373,"threads_started":1}
W20260620 14:57:05.418640 5131 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:46849 (0 consecutive failures): Network error: Failed to ping master at 127.21.159.254:46849: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
I20260620 14:57:05.419670 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:05.419339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:42437)
0620 14:57:05.419491 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:05.419562 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:57:05.420662 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:05.420302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:48981)
0620 14:57:05.420460 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:05.420539 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 14:57:05.420969 5131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:57:05.488643 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:05.487652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:50481)
0620 14:57:05.488211 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:05.488359 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":268,"threads_started":1}
I20260620 14:57:05.948558 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:05.947590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:43247)
0620 14:57:05.948117 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:05.948295 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":236,"threads_started":1}
I20260620 14:57:06.422403 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:06.421569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:58517)
0620 14:57:06.422057 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:06.422176 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":237,"threads_started":1}
I20260620 14:57:06.490681 5138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:06.489857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:52371)
0620 14:57:06.490404 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:06.490508 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":271,"threads_started":1}
I20260620 14:57:06.950472 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:06.949620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:48991)
0620 14:57:06.950145 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:06.950239 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":269,"threads_started":1}
I20260620 14:57:07.424221 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:07.423435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:47985)
0620 14:57:07.423950 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:07.424044 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":230,"threads_started":1}
I20260620 14:57:07.492424 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:07.491657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:46445)
0620 14:57:07.492102 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:07.492229 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260620 14:57:07.952284 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:07.951553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:44477)
0620 14:57:07.952024 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:07.952107 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
I20260620 14:57:08.426072 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:08.425239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:42135)
0620 14:57:08.425793 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:08.425886 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
I20260620 14:57:08.494306 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:08.493468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:46291)
0620 14:57:08.494040 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:08.494126 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":316,"threads_started":1}
I20260620 14:57:08.954073 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:08.953303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:53439)
0620 14:57:08.953784 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:08.953873 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":239,"threads_started":1}
I20260620 14:57:09.427691 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:09.426905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:60807)
0620 14:57:09.427436 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:09.427519 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":307,"threads_started":1}
I20260620 14:57:09.496166 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:09.495370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:60779)
0620 14:57:09.495906 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:09.495991 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":254,"threads_started":1}
I20260620 14:57:09.956224 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:09.955330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:52641)
0620 14:57:09.955926 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:09.956019 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":323,"threads_started":1}
I20260620 14:57:10.429399 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:10.428640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:57515)
0620 14:57:10.429123 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:10.429230 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":252,"threads_started":1}
I20260620 14:57:10.497962 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:10.497216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:55661)
0620 14:57:10.497697 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:10.497795 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260620 14:57:10.957975 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:10.957194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:41367)
0620 14:57:10.957682 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:10.957800 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260620 14:57:11.431013 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:11.430266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:57675)
0620 14:57:11.430740 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:11.430832 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":256,"threads_started":1}
I20260620 14:57:11.499835 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:11.498965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:51185)
0620 14:57:11.499498 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:11.499622 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":256,"threads_started":1}
I20260620 14:57:11.960908 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:11.959230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:41285)
0620 14:57:11.960427 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:11.960570 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":357,"threads_started":1}
I20260620 14:57:12.433275 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:12.432278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:48677)
0620 14:57:12.432832 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:12.432914 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":252,"threads_started":1}
I20260620 14:57:12.501560 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:12.500782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:53385)
0620 14:57:12.501291 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:12.501384 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":259,"threads_started":1}
I20260620 14:57:12.962859 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:12.962107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:59861)
0620 14:57:12.962594 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:12.962681 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":222,"threads_started":1}
I20260620 14:57:13.435124 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:13.434277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:51651)
0620 14:57:13.434772 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:13.434877 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260620 14:57:13.503321 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:13.502471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:50607)
0620 14:57:13.502981 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:13.503138 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
I20260620 14:57:13.964725 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:13.963890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:36629)
0620 14:57:13.964433 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:13.964528 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":256,"threads_started":1}
I20260620 14:57:14.436968 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:14.436119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:33885)
0620 14:57:14.436694 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:14.436784 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":280,"threads_started":1}
I20260620 14:57:14.505376 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:14.504542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:35987)
0620 14:57:14.505122 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:14.505206 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":360,"threads_started":1}
I20260620 14:57:14.966630 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:14.965793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:55683)
0620 14:57:14.966342 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:14.966437 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":261,"threads_started":1}
I20260620 14:57:15.438669 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:15.437951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:37665)
0620 14:57:15.438424 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:15.438509 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
I20260620 14:57:15.507210 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:15.506388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:53657)
0620 14:57:15.506836 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:15.506945 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20260620 14:57:15.968845 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:15.967793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:59763)
0620 14:57:15.968564 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:15.968659 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":327,"threads_started":1}
I20260620 14:57:16.440420 5167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:16.439693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:42443)
0620 14:57:16.440140 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:16.440242 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":216,"threads_started":1}
I20260620 14:57:16.509028 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:16.508262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:54183)
0620 14:57:16.508714 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:16.508846 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260620 14:57:16.970767 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:16.969952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:57353)
0620 14:57:16.970485 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:16.970596 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":282,"threads_started":1}
I20260620 14:57:17.442389 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:17.441579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:53447)
0620 14:57:17.442125 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:17.442224 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":323,"threads_started":1}
I20260620 14:57:17.511171 5171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:17.510222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:42821)
0620 14:57:17.510766 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:17.510876 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":270,"threads_started":1}
I20260620 14:57:17.972961 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:17.971886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:54083)
0620 14:57:17.972515 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:17.972715 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":323,"threads_started":1}
I20260620 14:57:18.444396 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:18.443599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:38759)
0620 14:57:18.444088 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:18.444191 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":236,"threads_started":1}
I20260620 14:57:18.512929 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:18.512156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:57881)
0620 14:57:18.512667 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:18.512753 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260620 14:57:18.974892 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:18.973995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:59067)
0620 14:57:18.974514 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:18.974657 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260620 14:57:19.446061 5176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:19.445323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:33887)
0620 14:57:19.445772 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:19.445871 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260620 14:57:19.514997 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:19.514171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.193:33999)
0620 14:57:19.514664 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:19.514796 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":255,"threads_started":1}
I20260620 14:57:19.976919 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:19.976106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.194:39835)
0620 14:57:19.976601 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:19.976718 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":237,"threads_started":1}
I20260620 14:57:20.422945 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 4707
I20260620 14:57:20.447860 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:20.424225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.21.159.193:43027)
0620 14:57:20.424921 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:20.424943 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:20.424997 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:57:20.426734 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:20.426746 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:20.427266 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:20.427581 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:20.429367 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:20.430635 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:20.432254 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:20.432958 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:20.433110 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:20.440725 (+ 7615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:20.440735 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:20.440752 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:20.440800 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:20.444284 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:20.445149 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:20.445156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:20.445163 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:20.445334 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:20.445706 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:20.445710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:20.445712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:20.446250 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:20.446486 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:20.447103 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 14:57:20.447575 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":814,"threads_started":1}
I20260620 14:57:20.448355 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:20.447425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46849 (local address 127.21.159.195:49103)
0620 14:57:20.447956 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:20.448157 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:46849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":273,"threads_started":1}
I20260620 14:57:20.449384 4864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:20.542662 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 4707
I20260620 14:57:20.560425 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 4872
I20260620 14:57:20.580158 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:20.561569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.21.159.194:42073)
0620 14:57:20.562144 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:20.562160 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:20.562201 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:57:20.563885 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:20.563896 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:20.564364 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:20.564755 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:20.566361 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:20.567784 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:20.569137 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:20.569866 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:20.570085 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:20.573010 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:20.573019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:20.573036 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:20.573080 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:20.576422 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:20.577223 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:20.577231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:20.577239 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:20.577444 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:20.577851 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:20.577857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:20.577860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:20.578462 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:20.578793 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:20.579332 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 14:57:20.579798 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":250,"threads_started":1}
I20260620 14:57:20.581786 4995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:20.669525 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 4872
I20260620 14:57:20.687574 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5003
I20260620 14:57:20.706544 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:20.688863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.21.159.195:33245)
0620 14:57:20.689510 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:20.689525 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:20.689568 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:57:20.691380 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:20.691390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:20.691842 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:20.692159 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:20.693650 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:20.694897 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:20.696286 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:20.696978 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:20.697127 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:20.699883 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:20.699891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:20.699909 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:20.699948 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:20.702936 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:20.703836 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:20.703844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:20.703851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:20.704042 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:20.704448 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:20.704451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:20.704453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:20.705062 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:20.705313 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:20.705735 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 14:57:20.706233 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":316,"threads_started":1}
I20260620 14:57:20.708093 5126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:20.792981 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5003
2026-06-20T14:57:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25118 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:57:20 dist-test-slave-3mqx krb5kdc[5189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:57:20 dist-test-slave-3mqx krb5kdc[5189](info): set up 2 sockets
Jun 20 14:57:20 dist-test-slave-3mqx krb5kdc[5189](info): commencing operation
krb5kdc: starting...
W20260620 14:57:21.912322 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:57:22 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:57:22.162701 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42219
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:22.530586 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:22.530941 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:22.531141 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:22.539381 5202 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:57:22.539485 5202 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:57:22.539551 5202 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:57:22.539628 5202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:22.539757 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:22.539865 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:22.539948 5202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:22.540028 5202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:22.550371 5202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:22.551831 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:22.553203 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:22.565992 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:22.565909 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:22.565929 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:22.566700 5202 server_base.cc:1034] running on GCE node
I20260620 14:57:22.567669 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:22.569229 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:22.570518 5202 hybrid_clock.cc:648] HybridClock initialized: now 1781967442570446 us; error 50 us; skew 500 ppm
I20260620 14:57:22.570977 5202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:22 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967442, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:22.574476 5202 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:57:22.576236 5202 webserver.cc:460] Webserver started at http://127.21.159.254:33601/ using document root <none> and password file <none>
I20260620 14:57:22.577051 5202 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:22.577201 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:22.577560 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:22.581029 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "26d4dea6f4974c75b5551530f43cbc29"
format_stamp: "Formatted at 2026-06-20 14:57:22 on dist-test-slave-3mqx"
server_key: "bf27f655a82ffc42d93dc70d6eaf6ff8"
server_key_iv: "cfd3860ee205fb59d915ad548d24ef52"
server_key_version: "encryptionkey@0"
I20260620 14:57:22.581945 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "26d4dea6f4974c75b5551530f43cbc29"
format_stamp: "Formatted at 2026-06-20 14:57:22 on dist-test-slave-3mqx"
server_key: "bf27f655a82ffc42d93dc70d6eaf6ff8"
server_key_iv: "cfd3860ee205fb59d915ad548d24ef52"
server_key_version: "encryptionkey@0"
I20260620 14:57:22.588030 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 14:57:22.592227 5217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:22.593762 5202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:57:22.593989 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "26d4dea6f4974c75b5551530f43cbc29"
format_stamp: "Formatted at 2026-06-20 14:57:22 on dist-test-slave-3mqx"
server_key: "bf27f655a82ffc42d93dc70d6eaf6ff8"
server_key_iv: "cfd3860ee205fb59d915ad548d24ef52"
server_key_version: "encryptionkey@0"
I20260620 14:57:22.594260 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:22.614710 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:22.619580 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:22.619902 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:22.640300 5202 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:42219
I20260620 14:57:22.640319 5268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:42219 every 8 connection(s)
I20260620 14:57:22.641881 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:22.645733 5269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:22.647292 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5202
I20260620 14:57:22.647537 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:57:22.648237 22143 external_mini_cluster.cc:1428] Setting key 950ddc7f8205d668f317ed27448545d2
I20260620 14:57:22.659333 5269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Bootstrap starting.
I20260620 14:57:22.662998 5269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:22.664091 5269 log.cc:824] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:57:22 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:22.667189 5269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: No bootstrap required, opened a new log
I20260620 14:57:22.673607 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:22.650517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.21.159.254:42219)
0620 14:57:22.651372 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:22.651388 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:22.651444 (+ 56us) server_negotiation.cc:407] Connection header received
0620 14:57:22.653817 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:22.653849 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:22.654371 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:22.654854 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:22.656488 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:22.658479 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:22.660219 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:22.661386 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:22.661607 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:22.666060 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:22.666067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:22.666082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:22.666122 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:22.669460 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:22.670209 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:22.670216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:22.670222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:22.670393 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:22.670810 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:22.670814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:22.670816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:22.671412 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:22.671853 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:22.672169 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 14:57:22.672592 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":351,"threads_started":1}
I20260620 14:57:22.676745 5269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:22.677256 5269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:22.677376 5269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d4dea6f4974c75b5551530f43cbc29, State: Initialized, Role: FOLLOWER
I20260620 14:57:22.677978 5269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:22.678290 5269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:22.678448 5269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:22.678665 5269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:22.681218 5269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:22.681820 5269 leader_election.cc:302] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d4dea6f4974c75b5551530f43cbc29; no voters:
I20260620 14:57:22.682453 5269 leader_election.cc:288] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:57:22.682561 5274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:22.683341 5274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 LEADER]: Becoming Leader. State: Replica: 26d4dea6f4974c75b5551530f43cbc29, State: Running, Role: LEADER
I20260620 14:57:22.683955 5274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:22.684741 5269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:22.686753 5275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d4dea6f4974c75b5551530f43cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } } }
I20260620 14:57:22.686890 5276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d4dea6f4974c75b5551530f43cbc29. Latest consensus state: current_term: 1 leader_uuid: "26d4dea6f4974c75b5551530f43cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } } }
I20260620 14:57:22.687235 5275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:22.687326 5276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:22.688586 5280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:22.695778 5280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:22.706197 5280 catalog_manager.cc:1347] Generated new cluster ID: 04b3aced9d1d4e44a4d9a7e9c6317f6a
I20260620 14:57:22.706349 5280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:22.746903 5280 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:57:22.748909 5280 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:22.759730 5280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Generated new TSK 0
I20260620 14:57:22.760622 5280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:57:22.830530 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42219
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:23.203611 5295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:23.203922 5295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:23.204098 5295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:23.211668 5295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:23.211804 5295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:23.212047 5295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:57:23.222443 5295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:23.223865 5295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:23.225278 5295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:23.239118 5303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:23.239177 5300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:23.239149 5301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:23.239910 5295 server_base.cc:1034] running on GCE node
I20260620 14:57:23.240543 5295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:23.241525 5295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:23.242743 5295 hybrid_clock.cc:648] HybridClock initialized: now 1781967443242694 us; error 36 us; skew 500 ppm
I20260620 14:57:23.243171 5295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:23 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967443, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:23.246485 5295 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:57:23.248139 5295 webserver.cc:460] Webserver started at http://127.21.159.193:37337/ using document root <none> and password file <none>
I20260620 14:57:23.248885 5295 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:23.249038 5295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:23.249377 5295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:23.252568 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "603af19a73ea4daeb9dceb0d9827dabc"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "6cca551ad4d32bc8e46dcd14f1a6dbfd"
server_key_iv: "8c06c215b6ca53a09bd9feb0e19de8e7"
server_key_version: "encryptionkey@0"
I20260620 14:57:23.253372 5295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "603af19a73ea4daeb9dceb0d9827dabc"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "6cca551ad4d32bc8e46dcd14f1a6dbfd"
server_key_iv: "8c06c215b6ca53a09bd9feb0e19de8e7"
server_key_version: "encryptionkey@0"
I20260620 14:57:23.258885 5295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:57:23.262653 5310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:23.264189 5295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:57:23.264398 5295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "603af19a73ea4daeb9dceb0d9827dabc"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "6cca551ad4d32bc8e46dcd14f1a6dbfd"
server_key_iv: "8c06c215b6ca53a09bd9feb0e19de8e7"
server_key_version: "encryptionkey@0"
I20260620 14:57:23.264626 5295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:23.282311 5295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:23.287532 5295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:23.287869 5295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:23.288916 5295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:23.290469 5295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:23.290599 5295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:23.290746 5295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:23.290810 5295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:23.326227 5295 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:40357
I20260620 14:57:23.326282 5422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:40357 every 8 connection(s)
I20260620 14:57:23.327644 5295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:23.331748 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5295
I20260620 14:57:23.332048 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:57:23.332909 22143 external_mini_cluster.cc:1428] Setting key 46e07f30fef901e2ce47e73edb8cf1d7
Jun 20 14:57:23 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967443, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:23.349524 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:23.330216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:57503 (local address 127.21.159.254:42219)
0620 14:57:23.330719 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:23.330724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:23.334369 (+ 3645us) server_negotiation.cc:407] Connection header received
0620 14:57:23.334426 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:23.334430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:23.334493 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:23.334614 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:23.336261 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.336973 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.338530 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.338940 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.339095 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:23.342624 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:23.342629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:23.342633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:23.342668 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:23.345670 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:23.346833 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:23.346837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:23.346840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:23.347006 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:23.347565 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:23.347572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:23.347580 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:23.347874 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:23.348039 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:23.348911 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 14:57:23.349151 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":237,"threads_started":1}
I20260620 14:57:23.350553 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:23.330831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:57503)
0620 14:57:23.331756 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:23.331818 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:23.334072 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:23.334874 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:23.334883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:23.335487 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:23.336079 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:23.336097 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.337134 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.337138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:23.338382 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:23.338394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.339196 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.339205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:23.339440 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:23.340127 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:23.340149 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:23.342358 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:23.345985 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:23.345996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:23.346025 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:23.346506 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:23.347189 (+ 683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:23.347195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:23.347199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:23.347426 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:23.348010 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:23.348019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:23.348499 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 14:57:23.348776 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 14:57:23.349362 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":370,"threads_started":1}
I20260620 14:57:23.352403 5423 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42219
I20260620 14:57:23.352744 5423 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:23.353492 5423 heartbeater.cc:511] Master 127.21.159.254:42219 requested a full tablet report, sending...
I20260620 14:57:23.356228 5234 ts_manager.cc:194] Registered new tserver with Master: 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357)
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:57:23.358179 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:57503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:57:23.390991 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42219
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:23.732213 5429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:23.732517 5429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:23.732620 5429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:23.740031 5429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:23.740155 5429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:23.740269 5429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:57:23.750298 5429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:23.751706 5429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:23.753050 5429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:23.766993 5435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:23.767086 5434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:23.767006 5437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:23.768095 5429 server_base.cc:1034] running on GCE node
I20260620 14:57:23.768844 5429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:23.769922 5429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:23.771183 5429 hybrid_clock.cc:648] HybridClock initialized: now 1781967443771096 us; error 59 us; skew 500 ppm
I20260620 14:57:23.771554 5429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:23 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967443, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:23.775249 5429 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:57:23.776814 5429 webserver.cc:460] Webserver started at http://127.21.159.194:44175/ using document root <none> and password file <none>
I20260620 14:57:23.777498 5429 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:23.777598 5429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:23.777904 5429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:23.780941 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "e4cb01e03e3f479d97a242f00b4ca9e3"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "011ff8d143ceb974c83eb3b52413211f"
server_key_iv: "08cfe11ecbbd771819a251a7254d9dc6"
server_key_version: "encryptionkey@0"
I20260620 14:57:23.781687 5429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "e4cb01e03e3f479d97a242f00b4ca9e3"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "011ff8d143ceb974c83eb3b52413211f"
server_key_iv: "08cfe11ecbbd771819a251a7254d9dc6"
server_key_version: "encryptionkey@0"
I20260620 14:57:23.787388 5429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:57:23.791163 5444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:23.792586 5429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:57:23.792722 5429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "e4cb01e03e3f479d97a242f00b4ca9e3"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "011ff8d143ceb974c83eb3b52413211f"
server_key_iv: "08cfe11ecbbd771819a251a7254d9dc6"
server_key_version: "encryptionkey@0"
I20260620 14:57:23.792963 5429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:23.819020 5429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:23.824066 5429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:23.824535 5429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:23.825732 5429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:23.827292 5429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:23.827404 5429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:23.827572 5429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:23.827656 5429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:23.864893 5429 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:40283
I20260620 14:57:23.864987 5556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:40283 every 8 connection(s)
I20260620 14:57:23.866334 5429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:57:23.876190 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5429
I20260620 14:57:23.876381 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:57:23.876861 22143 external_mini_cluster.cc:1428] Setting key 2b35d2fb69e4935ee214999f0e390b35
Jun 20 14:57:23 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967443, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:23.889495 5559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:23.869207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:55051 (local address 127.21.159.254:42219)
0620 14:57:23.869660 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:23.869665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:23.870641 (+ 976us) server_negotiation.cc:407] Connection header received
0620 14:57:23.872780 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:23.872787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:23.872882 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:23.873026 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:23.874627 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.875544 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.877385 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.877951 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.878094 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:23.882655 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:23.882660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:23.882664 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:23.882698 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:23.885489 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:23.886546 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:23.886553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:23.886558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:23.886739 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:23.887335 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:23.887343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:23.887348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:23.887745 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:23.887911 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:23.888693 (+ 782us) server_negotiation.cc:299] Negotiation successful
0620 14:57:23.888972 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260620 14:57:23.890360 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:23.869748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:55051)
0620 14:57:23.870467 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:23.870512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:23.872481 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:23.873286 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:23.873295 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:23.873840 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:23.874445 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:23.874462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.875706 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.875715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:23.877234 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:23.877248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:23.878207 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:23.878217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:23.878465 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:23.879232 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:23.879262 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:23.882379 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:23.885640 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:23.885653 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:23.885674 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:23.886400 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:23.886889 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:23.886893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:23.886896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:23.887211 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:23.887887 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:23.887894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:23.888288 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 14:57:23.888543 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 14:57:23.889022 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":271,"threads_started":1}
I20260620 14:57:23.892174 5557 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42219
I20260620 14:57:23.892525 5557 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:23.893266 5557 heartbeater.cc:511] Master 127.21.159.254:42219 requested a full tablet report, sending...
I20260620 14:57:23.895165 5234 ts_manager.cc:194] Registered new tserver with Master: e4cb01e03e3f479d97a242f00b4ca9e3 (127.21.159.194:40283)
I20260620 14:57:23.896090 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:55051
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:57:23.932279 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42219
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:24.269094 5563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:24.269495 5563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:24.269693 5563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:24.277882 5563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:24.278004 5563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:24.278206 5563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:57:24.288370 5563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:24.289872 5563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:24.291527 5563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:24.305478 5568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:24.305445 5571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:24.305521 5569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:24.306473 5563 server_base.cc:1034] running on GCE node
I20260620 14:57:24.307227 5563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:24.308310 5563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:24.309546 5563 hybrid_clock.cc:648] HybridClock initialized: now 1781967444309484 us; error 53 us; skew 500 ppm
I20260620 14:57:24.309927 5563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:24 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:24.313467 5563 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:57:24.315007 5563 webserver.cc:460] Webserver started at http://127.21.159.195:36439/ using document root <none> and password file <none>
I20260620 14:57:24.315788 5563 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:24.315897 5563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:24.316180 5563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:57:24.319567 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "af37ae04112f4f4fa0e97f545aae4817"
format_stamp: "Formatted at 2026-06-20 14:57:24 on dist-test-slave-3mqx"
server_key: "49a50d393d8a370431f6fcfa13d52d43"
server_key_iv: "1f1624585790a5386ef0011c0256e025"
server_key_version: "encryptionkey@0"
I20260620 14:57:24.320363 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "af37ae04112f4f4fa0e97f545aae4817"
format_stamp: "Formatted at 2026-06-20 14:57:24 on dist-test-slave-3mqx"
server_key: "49a50d393d8a370431f6fcfa13d52d43"
server_key_iv: "1f1624585790a5386ef0011c0256e025"
server_key_version: "encryptionkey@0"
I20260620 14:57:24.326295 5563 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260620 14:57:24.330927 5578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:24.332508 5563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:57:24.332684 5563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "af37ae04112f4f4fa0e97f545aae4817"
format_stamp: "Formatted at 2026-06-20 14:57:24 on dist-test-slave-3mqx"
server_key: "49a50d393d8a370431f6fcfa13d52d43"
server_key_iv: "1f1624585790a5386ef0011c0256e025"
server_key_version: "encryptionkey@0"
I20260620 14:57:24.332864 5563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:24.344406 5563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:24.349334 5563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:24.349696 5563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:24.350803 5563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:24.352466 5563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:24.352550 5563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:24.352707 5563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:24.352766 5563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:24.362143 5423 heartbeater.cc:503] Master 127.21.159.254:42219 was elected leader, sending a full tablet report...
I20260620 14:57:24.389408 5563 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:34179
I20260620 14:57:24.389446 5690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:34179 every 8 connection(s)
I20260620 14:57:24.391166 5563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:24.398468 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5563
I20260620 14:57:24.398694 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:57:24.399362 22143 external_mini_cluster.cc:1428] Setting key 638f271317a01d2e1bdcd6d039ff0769
Jun 20 14:57:24 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967444, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:24.415392 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.393984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:37793 (local address 127.21.159.254:42219)
0620 14:57:24.396933 (+ 2949us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.396940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.396968 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:57:24.400123 (+ 3155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.400130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.400231 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.400396 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:24.402346 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.403357 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.404992 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.405448 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.405608 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.409280 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:24.409285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:24.409288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:24.409322 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:24.411867 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:24.412585 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:24.412593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:24.412600 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:24.412716 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:24.413158 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:24.413164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:24.413167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:24.413508 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:24.413666 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.414725 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.414922 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2795,"thread_start_us":2648,"threads_started":1}
I20260620 14:57:24.416368 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.394721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:37793)
0620 14:57:24.396283 (+ 1562us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:24.396378 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:24.399744 (+ 3366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:24.400683 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:24.400693 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:24.401502 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:24.402149 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.402169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.403519 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.403526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.404823 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.404837 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.405747 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.405755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.406009 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.406740 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:24.406763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:24.409059 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:24.412004 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:24.412013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:24.412030 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:24.412455 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:24.412844 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:24.412849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:24.412853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:24.413042 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:24.413781 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:24.413790 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:24.414243 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 14:57:24.414567 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 14:57:24.415132 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":593,"threads_started":1}
I20260620 14:57:24.417763 5691 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42219
I20260620 14:57:24.418068 5691 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:24.418746 5691 heartbeater.cc:511] Master 127.21.159.254:42219 requested a full tablet report, sending...
I20260620 14:57:24.420204 5234 ts_manager.cc:194] Registered new tserver with Master: af37ae04112f4f4fa0e97f545aae4817 (127.21.159.195:34179)
I20260620 14:57:24.420986 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:37793
I20260620 14:57:24.427824 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:57:24.445294 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.432458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.21.159.254:42219)
0620 14:57:24.432640 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.432644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.433076 (+ 432us) server_negotiation.cc:407] Connection header received
0620 14:57:24.433226 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.433230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.433297 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.433387 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:24.434871 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.435758 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.437137 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.437605 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.437698 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.440200 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:24.440205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:24.440208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:24.440242 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:24.442875 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:24.443578 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:24.443581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:24.443584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:24.443702 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:24.444044 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:24.444047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:24.444049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:24.444372 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:24.444550 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.444902 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.445070 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:57:24.449399 5234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:24.456234 5234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:24.482144 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.472786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.21.159.193:40357)
0620 14:57:24.473674 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.473682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.474550 (+ 868us) server_negotiation.cc:407] Connection header received
0620 14:57:24.474744 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.474752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.474928 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.475145 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:24.475639 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.477134 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.478985 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.480524 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.480723 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.480921 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.481661 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.481828 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":551,"threads_started":1}
I20260620 14:57:24.482183 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.469183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.21.159.195:34179)
0620 14:57:24.470615 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.470628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.471636 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 14:57:24.472494 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.472503 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.472794 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.472970 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:24.474287 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.476086 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.479435 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.480652 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.480846 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.481049 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.481656 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.481832 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":984,"threads_started":1}
I20260620 14:57:24.482223 5706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.469044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:34179 (local address 127.0.0.1:54802)
0620 14:57:24.471435 (+ 2391us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:24.471487 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:24.471838 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:24.473389 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:24.473396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:24.473426 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:24.474037 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.474061 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.476278 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.476282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.479242 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.479258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.481182 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.481190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.481403 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.481455 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:57:24.481601 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:57:24.481801 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2073,"mutex_wait_us":135,"thread_start_us":3193,"threads_started":1}
I20260620 14:57:24.482409 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.469837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.21.159.194:40283)
0620 14:57:24.472195 (+ 2358us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.472206 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.472790 (+ 584us) server_negotiation.cc:407] Connection header received
0620 14:57:24.472894 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.472904 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.473145 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.473364 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:24.474537 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.476125 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.479420 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.480870 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.481053 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.481197 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.481916 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.482077 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2190,"thread_start_us":2059,"threads_started":1}
I20260620 14:57:24.482947 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.469368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:40283 (local address 127.0.0.1:46808)
0620 14:57:24.472508 (+ 3140us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:24.472525 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:24.472647 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:24.473630 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:24.473638 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:24.473665 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:24.474325 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.474340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.476283 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.476287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.479244 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.479261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.481358 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.481365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.481547 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.481590 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:57:24.481664 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:57:24.481812 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3013,"mutex_wait_us":193,"thread_start_us":1574,"threads_started":1}
I20260620 14:57:24.485010 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.473660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:40357 (local address 127.0.0.1:58766)
0620 14:57:24.474452 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:24.474470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:24.474582 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:24.475119 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:24.475124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:24.475148 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:24.475497 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.475505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.477275 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.477282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.478831 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.478843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.481050 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.481056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.481285 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.481456 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 14:57:24.484498 (+ 3042us) client_negotiation.cc:240] Negotiation successful
0620 14:57:24.484651 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":53,"thread_start_us":423,"threads_started":1}
I20260620 14:57:24.486835 5492 tablet_service.cc:1467] Processing CreateTablet for tablet 4901759c559a4fc1984a1d66cac70803 (DEFAULT_TABLE table=test-table [id=a4e01db3b79f4d85a0b6e851ffabc00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:24.487119 5626 tablet_service.cc:1467] Processing CreateTablet for tablet 4901759c559a4fc1984a1d66cac70803 (DEFAULT_TABLE table=test-table [id=a4e01db3b79f4d85a0b6e851ffabc00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:24.488210 5492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4901759c559a4fc1984a1d66cac70803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:24.488595 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4901759c559a4fc1984a1d66cac70803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:24.488431 5358 tablet_service.cc:1467] Processing CreateTablet for tablet 4901759c559a4fc1984a1d66cac70803 (DEFAULT_TABLE table=test-table [id=a4e01db3b79f4d85a0b6e851ffabc00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:24.489723 5358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4901759c559a4fc1984a1d66cac70803. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:24.503836 5710 tablet_bootstrap.cc:492] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Bootstrap starting.
I20260620 14:57:24.507297 5710 tablet_bootstrap.cc:654] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:24.507912 5712 tablet_bootstrap.cc:492] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Bootstrap starting.
I20260620 14:57:24.508774 5711 tablet_bootstrap.cc:492] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Bootstrap starting.
I20260620 14:57:24.510202 5710 log.cc:824] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:24.512718 5711 tablet_bootstrap.cc:654] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:24.513172 5710 tablet_bootstrap.cc:492] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: No bootstrap required, opened a new log
I20260620 14:57:24.513470 5710 ts_tablet_manager.cc:1397] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 14:57:24.514017 5711 log.cc:824] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:24.514526 5712 tablet_bootstrap.cc:654] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:24.515892 5712 log.cc:824] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:24.517184 5711 tablet_bootstrap.cc:492] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: No bootstrap required, opened a new log
I20260620 14:57:24.517494 5711 ts_tablet_manager.cc:1397] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:57:24.519083 5712 tablet_bootstrap.cc:492] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: No bootstrap required, opened a new log
I20260620 14:57:24.519402 5712 ts_tablet_manager.cc:1397] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260620 14:57:24.522065 5710 raft_consensus.cc:348] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.522447 5710 raft_consensus.cc:374] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:24.522548 5710 raft_consensus.cc:729] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cb01e03e3f479d97a242f00b4ca9e3, State: Initialized, Role: FOLLOWER
I20260620 14:57:24.523170 5710 consensus_queue.cc:260] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.524339 5557 heartbeater.cc:503] Master 127.21.159.254:42219 was elected leader, sending a full tablet report...
I20260620 14:57:24.525125 5710 ts_tablet_manager.cc:1428] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260620 14:57:24.529337 5712 raft_consensus.cc:348] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.529572 5711 raft_consensus.cc:348] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.529896 5712 raft_consensus.cc:374] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:24.530012 5712 raft_consensus.cc:729] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af37ae04112f4f4fa0e97f545aae4817, State: Initialized, Role: FOLLOWER
I20260620 14:57:24.530068 5711 raft_consensus.cc:374] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:24.530223 5711 raft_consensus.cc:729] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603af19a73ea4daeb9dceb0d9827dabc, State: Initialized, Role: FOLLOWER
I20260620 14:57:24.530712 5712 consensus_queue.cc:260] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.530979 5711 consensus_queue.cc:260] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.531865 5691 heartbeater.cc:503] Master 127.21.159.254:42219 was elected leader, sending a full tablet report...
I20260620 14:57:24.532614 5712 ts_tablet_manager.cc:1428] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260620 14:57:24.534268 5711 ts_tablet_manager.cc:1428] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260620 14:57:24.580178 5424 tablet.cc:2369] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:24.618892 5558 tablet.cc:2369] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:24.642997 5692 tablet.cc:2369] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:57:24.819772 5717 raft_consensus.cc:482] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:24.820058 5717 raft_consensus.cc:504] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.821730 5717 leader_election.cc:288] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357), e4cb01e03e3f479d97a242f00b4ca9e3 (127.21.159.194:40283)
I20260620 14:57:24.828315 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.822361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:42247 (local address 127.21.159.193:40357)
0620 14:57:24.822659 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.822667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.822692 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:57:24.822960 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.822972 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.823039 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.823217 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:24.823674 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.824453 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.826304 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.827310 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.827492 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.827588 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.827892 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.828031 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:57:24.828423 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.822169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:40357 (local address 127.21.159.195:42247)
0620 14:57:24.822598 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:24.822621 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:24.822783 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:24.823240 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:24.823245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:24.823274 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:24.823512 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.823522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.824616 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.824622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.826126 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.826139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.827476 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.827483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.827709 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.827738 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:57:24.827865 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:57:24.828016 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":25,"spinlock_wait_cycles":70656}
I20260620 14:57:24.828462 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.822274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:54801 (local address 127.21.159.194:40283)
0620 14:57:24.822517 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.822525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.823329 (+ 804us) server_negotiation.cc:407] Connection header received
0620 14:57:24.823582 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.823586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.823647 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.823747 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:24.824280 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.824966 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.826698 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.827536 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.827717 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.827789 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.828067 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.828204 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:57:24.828634 5719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.822361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:40283 (local address 127.21.159.195:54801)
0620 14:57:24.823200 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:24.823218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:24.823411 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:24.823760 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:24.823764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:24.823786 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:24.824102 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.824111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.825119 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.825124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.826535 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:24.826546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.827740 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.827746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:24.827907 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.827929 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:57:24.828029 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:57:24.828153 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":464,"threads_started":1}
I20260620 14:57:24.829681 5378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4901759c559a4fc1984a1d66cac70803" candidate_uuid: "af37ae04112f4f4fa0e97f545aae4817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603af19a73ea4daeb9dceb0d9827dabc" is_pre_election: true
I20260620 14:57:24.829905 5512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4901759c559a4fc1984a1d66cac70803" candidate_uuid: "af37ae04112f4f4fa0e97f545aae4817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" is_pre_election: true
I20260620 14:57:24.830273 5378 raft_consensus.cc:2454] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af37ae04112f4f4fa0e97f545aae4817 in term 0.
I20260620 14:57:24.830590 5512 raft_consensus.cc:2454] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af37ae04112f4f4fa0e97f545aae4817 in term 0.
I20260620 14:57:24.831135 5582 leader_election.cc:302] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603af19a73ea4daeb9dceb0d9827dabc, af37ae04112f4f4fa0e97f545aae4817; no voters:
I20260620 14:57:24.831625 5717 raft_consensus.cc:2781] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:24.831782 5717 raft_consensus.cc:482] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:24.831892 5717 raft_consensus.cc:3037] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:24.834033 5717 raft_consensus.cc:504] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.834751 5717 leader_election.cc:288] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [CANDIDATE]: Term 1 election: Requested vote from peers 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357), e4cb01e03e3f479d97a242f00b4ca9e3 (127.21.159.194:40283)
I20260620 14:57:24.835285 5378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4901759c559a4fc1984a1d66cac70803" candidate_uuid: "af37ae04112f4f4fa0e97f545aae4817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603af19a73ea4daeb9dceb0d9827dabc"
I20260620 14:57:24.835417 5512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4901759c559a4fc1984a1d66cac70803" candidate_uuid: "af37ae04112f4f4fa0e97f545aae4817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3"
I20260620 14:57:24.835521 5378 raft_consensus.cc:3037] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:24.835781 5512 raft_consensus.cc:3037] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:24.838181 5378 raft_consensus.cc:2454] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af37ae04112f4f4fa0e97f545aae4817 in term 1.
I20260620 14:57:24.838572 5582 leader_election.cc:302] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603af19a73ea4daeb9dceb0d9827dabc, af37ae04112f4f4fa0e97f545aae4817; no voters:
I20260620 14:57:24.838673 5512 raft_consensus.cc:2454] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af37ae04112f4f4fa0e97f545aae4817 in term 1.
I20260620 14:57:24.838847 5717 raft_consensus.cc:2781] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:24.839341 5717 raft_consensus.cc:686] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 1 LEADER]: Becoming Leader. State: Replica: af37ae04112f4f4fa0e97f545aae4817, State: Running, Role: LEADER
I20260620 14:57:24.839843 5717 consensus_queue.cc:237] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } }
I20260620 14:57:24.844547 5233 catalog_manager.cc:5495] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 reported cstate change: term changed from 0 to 1, leader changed from <none> to af37ae04112f4f4fa0e97f545aae4817 (127.21.159.195). New cstate: current_term: 1 leader_uuid: "af37ae04112f4f4fa0e97f545aae4817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } health_report { overall_health: UNKNOWN } } }
I20260620 14:57:24.937768 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.932390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.21.159.195:34179)
0620 14:57:24.932571 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.932575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.932592 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:57:24.932778 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.932782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.932859 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.932982 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:24.933335 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.934045 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.935554 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:24.935997 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:24.936124 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:24.936637 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:24.937197 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:24.937314 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:24.937382 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:57:24.937449 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:57:24.946135 5378 raft_consensus.cc:1261] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Refusing update from remote peer af37ae04112f4f4fa0e97f545aae4817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:24.946135 5512 raft_consensus.cc:1261] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 FOLLOWER]: Refusing update from remote peer af37ae04112f4f4fa0e97f545aae4817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:24.947371 5717 consensus_queue.cc:1035] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:24.947763 5720 consensus_queue.cc:1035] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:24.962103 5725 mvcc.cc:205] Tried to move back new op lower bound from 7298938654488973312 to 7298938654068224000. Current Snapshot: MvccSnapshot[applied={T|T < 7298938654488973312}]
I20260620 14:57:24.963140 5723 mvcc.cc:205] Tried to move back new op lower bound from 7298938654488973312 to 7298938654068224000. Current Snapshot: MvccSnapshot[applied={T|T < 7298938654488973312}]
I20260620 14:57:24.979839 5233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:24.980189 5233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:24.983975 5233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Sending DeleteTablet for 3 replicas of tablet 4901759c559a4fc1984a1d66cac70803
I20260620 14:57:24.985350 5492 tablet_service.cc:1514] Processing DeleteTablet for tablet 4901759c559a4fc1984a1d66cac70803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:24 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:46808
I20260620 14:57:24.985342 5358 tablet_service.cc:1514] Processing DeleteTablet for tablet 4901759c559a4fc1984a1d66cac70803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:24 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:58766
I20260620 14:57:24.985509 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5295
I20260620 14:57:24.986353 5626 tablet_service.cc:1514] Processing DeleteTablet for tablet 4901759c559a4fc1984a1d66cac70803 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:24 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:54802
I20260620 14:57:25.001705 5733 tablet_replica.cc:330] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: stopping tablet replica
I20260620 14:57:25.002168 5733 raft_consensus.cc:2229] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:25.002681 5733 raft_consensus.cc:2258] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:25.007402 5733 ts_tablet_manager.cc:1905] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 14:57:25 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:57:25.011600 5732 tablet_replica.cc:330] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: stopping tablet replica
I20260620 14:57:25.012245 5735 tablet_replica.cc:330] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: stopping tablet replica
I20260620 14:57:25.012965 5735 raft_consensus.cc:2229] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:25.013510 5732 raft_consensus.cc:2229] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:25.014150 5732 raft_consensus.cc:2258] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:25.014899 5735 raft_consensus.cc:2258] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:25.015734 5733 ts_tablet_manager.cc:1918] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:25.015966 5733 log.cc:1196] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/4901759c559a4fc1984a1d66cac70803
I20260620 14:57:25.016623 5733 ts_tablet_manager.cc:1939] T 4901759c559a4fc1984a1d66cac70803 P 603af19a73ea4daeb9dceb0d9827dabc: Deleting consensus metadata
I20260620 14:57:25.017498 5732 ts_tablet_manager.cc:1905] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:25.018679 5221 catalog_manager.cc:4841] TS 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357): tablet 4901759c559a4fc1984a1d66cac70803 (table test-table [id=a4e01db3b79f4d85a0b6e851ffabc00f]) successfully deleted
I20260620 14:57:25.019588 5735 ts_tablet_manager.cc:1905] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:25.020839 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:24.986935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.21.159.193:40357)
0620 14:57:24.987196 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:24.987206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:24.997094 (+ 9888us) server_negotiation.cc:407] Connection header received
0620 14:57:24.997800 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:24.997811 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:24.997950 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:24.998050 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:25.003193 (+ 5143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:25.003983 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:25.005432 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:25.006073 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:25.006207 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:25.012564 (+ 6357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:25.012580 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:25.012601 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:25.012672 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:25.017002 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:25.018105 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:25.018115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:25.018118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:25.018285 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:25.018787 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:25.018793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:25.018796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:25.019272 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:25.019479 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:25.020071 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 14:57:25.020304 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:57:25.021739 5418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:25.027215 5735 ts_tablet_manager.cc:1918] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:25.027583 5735 log.cc:1196] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/4901759c559a4fc1984a1d66cac70803
I20260620 14:57:25.028210 5735 ts_tablet_manager.cc:1939] T 4901759c559a4fc1984a1d66cac70803 P af37ae04112f4f4fa0e97f545aae4817: Deleting consensus metadata
I20260620 14:57:25.029542 5732 ts_tablet_manager.cc:1918] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:25.029816 5732 log.cc:1196] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/4901759c559a4fc1984a1d66cac70803
I20260620 14:57:25.030347 5221 catalog_manager.cc:4841] TS af37ae04112f4f4fa0e97f545aae4817 (127.21.159.195:34179): tablet 4901759c559a4fc1984a1d66cac70803 (table test-table [id=a4e01db3b79f4d85a0b6e851ffabc00f]) successfully deleted
I20260620 14:57:25.030519 5732 ts_tablet_manager.cc:1939] T 4901759c559a4fc1984a1d66cac70803 P e4cb01e03e3f479d97a242f00b4ca9e3: Deleting consensus metadata
I20260620 14:57:25.032841 5220 catalog_manager.cc:4841] TS e4cb01e03e3f479d97a242f00b4ca9e3 (127.21.159.194:40283): tablet 4901759c559a4fc1984a1d66cac70803 (table test-table [id=a4e01db3b79f4d85a0b6e851ffabc00f]) successfully deleted
I20260620 14:57:25.127274 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5295
I20260620 14:57:25.146435 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5429
Jun 20 14:57:25 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:57:25.165108 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:25.147367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.21.159.194:40283)
0620 14:57:25.147584 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:25.147591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:25.147613 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:57:25.147706 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:25.147712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:25.147795 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:25.147906 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:25.149436 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:25.150564 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:25.152246 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:25.152890 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:25.153023 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:25.157753 (+ 4730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:25.157775 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:25.157797 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:25.157854 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:25.161790 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:25.162623 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:25.162633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:25.162637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:25.162804 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:25.163370 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:25.163378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:25.163380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:25.163800 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:25.164117 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:25.164520 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 14:57:25.164761 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:57:25.165882 5552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:25.257318 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5429
I20260620 14:57:25.274668 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5563
Jun 20 14:57:25 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:57:25.291572 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:25.275539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.21.159.195:34179)
0620 14:57:25.275727 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:25.275735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:25.275759 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:57:25.275973 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:25.275979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:25.276061 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:25.276177 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:25.277477 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:25.278313 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:25.279756 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:25.280235 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:25.280377 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:25.285067 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:25.285081 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:25.285097 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:25.285157 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:25.288555 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:25.289295 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:25.289311 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:25.289314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:25.289474 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:25.289905 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:25.289910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:25.289913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:25.290265 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:25.290622 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:25.290918 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:57:25.291177 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:57:25.292393 5686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:25.427568 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5563
I20260620 14:57:25.450405 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5202
I20260620 14:57:25.451746 5264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:25.565546 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5202
I20260620 14:57:45.587642 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42219
--webserver_interface=127.21.159.254
--webserver_port=33601
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:57:45.927124 5740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:45.927531 5740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:45.927692 5740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:45.936203 5740 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:57:45.936348 5740 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:57:45.936422 5740 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:57:45.936496 5740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:57:45.936578 5740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:45.936683 5740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:45.936756 5740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:57:45.936841 5740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:57:45.947186 5740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=33601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:45.948586 5740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:45.950008 5740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:45.974406 5746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:45.974434 5749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:45.974465 5747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:45.975380 5740 server_base.cc:1034] running on GCE node
I20260620 14:57:45.976500 5740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:45.978399 5740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:45.979708 5740 hybrid_clock.cc:648] HybridClock initialized: now 1781967465979634 us; error 52 us; skew 500 ppm
I20260620 14:57:45.980183 5740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:45 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967465, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:45.984418 5740 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:57:45.986274 5740 webserver.cc:460] Webserver started at http://127.21.159.254:33601/ using document root <none> and password file <none>
I20260620 14:57:45.987181 5740 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:45.987308 5740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:45.994854 5740 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:57:45.998416 5756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.000295 5740 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:57:46.000499 5740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "26d4dea6f4974c75b5551530f43cbc29"
format_stamp: "Formatted at 2026-06-20 14:57:22 on dist-test-slave-3mqx"
server_key: "bf27f655a82ffc42d93dc70d6eaf6ff8"
server_key_iv: "cfd3860ee205fb59d915ad548d24ef52"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.001657 5740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:46.036540 5740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:46.041535 5740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:46.041822 5740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:46.062546 5740 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:42219
I20260620 14:57:46.062568 5807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:42219 every 8 connection(s)
I20260620 14:57:46.064286 5740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:57:46.071806 5808 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:57:46.073647 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5740
I20260620 14:57:46.074640 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:40357
--local_ip_for_outbound_sockets=127.21.159.193
--tserver_master_addrs=127.21.159.254:42219
--webserver_port=37337
--webserver_interface=127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:57:46.076525 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Bootstrap starting.
I20260620 14:57:46.104250 5808 log.cc:824] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:46.120371 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:57:46.121089 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Bootstrap complete.
I20260620 14:57:46.135252 5808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:46.136287 5808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d4dea6f4974c75b5551530f43cbc29, State: Initialized, Role: FOLLOWER
I20260620 14:57:46.137109 5808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:46.137396 5808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:57:46.137538 5808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:57:46.137753 5808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:57:46.141968 5808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:46.142553 5808 leader_election.cc:302] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d4dea6f4974c75b5551530f43cbc29; no voters:
I20260620 14:57:46.143270 5808 leader_election.cc:288] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:57:46.143500 5812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:57:46.144747 5812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 LEADER]: Becoming Leader. State: Replica: 26d4dea6f4974c75b5551530f43cbc29, State: Running, Role: LEADER
I20260620 14:57:46.145455 5812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:57:46.145750 5808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:57:46.148129 5814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d4dea6f4974c75b5551530f43cbc29. Latest consensus state: current_term: 2 leader_uuid: "26d4dea6f4974c75b5551530f43cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } } }
I20260620 14:57:46.148523 5814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:46.149008 5813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "26d4dea6f4974c75b5551530f43cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } } }
I20260620 14:57:46.149310 5813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:57:46.154134 5817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:57:46.165261 5817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:57:46.168506 5817 catalog_manager.cc:1259] Loaded cluster ID: 04b3aced9d1d4e44a4d9a7e9c6317f6a
I20260620 14:57:46.168737 5817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:57:46.174275 5817 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:57:46.176590 5817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Loaded TSK: 0
I20260620 14:57:46.193434 5830 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:57:46.211445 5817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Generated new TSK 1
I20260620 14:57:46.211869 5817 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:57:46.477744 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:46.478087 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:46.478253 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:46.485882 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:46.486050 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:46.486233 5810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:57:46.496503 5810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:40357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=37337
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:46.498000 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:46.499648 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:46.513898 5837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:46.513999 5836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:46.516516 5839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:46.517400 5810 server_base.cc:1034] running on GCE node
I20260620 14:57:46.518232 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:46.519559 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:46.520843 5810 hybrid_clock.cc:648] HybridClock initialized: now 1781967466520787 us; error 45 us; skew 500 ppm
I20260620 14:57:46.521309 5810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:46 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:46.525501 5810 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:57:46.527789 5810 webserver.cc:460] Webserver started at http://127.21.159.193:37337/ using document root <none> and password file <none>
I20260620 14:57:46.528677 5810 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:46.528867 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:46.535894 5810 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:57:46.539377 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.541143 5810 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:57:46.541354 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "603af19a73ea4daeb9dceb0d9827dabc"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "6cca551ad4d32bc8e46dcd14f1a6dbfd"
server_key_iv: "8c06c215b6ca53a09bd9feb0e19de8e7"
server_key_version: "encryptionkey@0"
I20260620 14:57:46.542559 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:46.562261 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:46.567871 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:46.568266 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:46.569542 5810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:46.571214 5810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:46.571324 5810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.571414 5810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:46.571487 5810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:46.608093 5810 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:40357
I20260620 14:57:46.608145 5958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:40357 every 8 connection(s)
I20260620 14:57:46.609676 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:57:46.617520 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5810
I20260620 14:57:46.618304 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:40283
--local_ip_for_outbound_sockets=127.21.159.194
--tserver_master_addrs=127.21.159.254:42219
--webserver_port=44175
--webserver_interface=127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:57:46 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:46.635816 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.613065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:45467)
0620 14:57:46.613896 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:46.613953 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:46.615919 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:46.617637 (+ 1718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:46.617649 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:46.618324 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:46.618995 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.619017 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.620767 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.620787 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.622094 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:46.622107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.623208 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.623215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:46.623501 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.624299 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:46.624324 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:46.627026 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:46.631098 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.631109 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.631126 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.631585 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.632073 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:46.632078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:46.632081 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:46.632293 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:46.633102 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:46.633111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:46.633478 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 14:57:46.633755 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 14:57:46.634375 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":369,"threads_started":1}
I20260620 14:57:46.635816 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:46.612916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:45467 (local address 127.21.159.254:42219)
0620 14:57:46.613579 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:46.613592 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:46.614139 (+ 547us) server_negotiation.cc:407] Connection header received
0620 14:57:46.616574 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:46.616594 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:46.617039 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:46.617461 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:46.619310 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.620582 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.622264 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:46.622932 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:46.623183 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:46.627443 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:46.627453 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:46.627470 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:46.627520 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:46.630880 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.631755 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.631764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.631769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.631939 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:46.632425 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:46.632429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:46.632431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:46.632944 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:46.633164 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:46.633987 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 14:57:46.634377 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":319,"threads_started":1}
I20260620 14:57:46.639444 5959 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42219
I20260620 14:57:46.639830 5959 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:46.640856 5959 heartbeater.cc:511] Master 127.21.159.254:42219 requested a full tablet report, sending...
I20260620 14:57:46.643594 5773 ts_manager.cc:194] Registered new tserver with Master: 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357)
I20260620 14:57:46.645068 5773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:45467
W20260620 14:57:46.955664 5963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:46.955955 5963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:46.956115 5963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:46.963845 5963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:46.963968 5963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:46.964160 5963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:57:46.974256 5963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:40283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=44175
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:46.975677 5963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:46.977012 5963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:46.990643 5968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:46.990751 5969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:46.990767 5971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:46.992082 5963 server_base.cc:1034] running on GCE node
I20260620 14:57:46.992905 5963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:46.994354 5963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:46.995615 5963 hybrid_clock.cc:648] HybridClock initialized: now 1781967466995530 us; error 72 us; skew 500 ppm
I20260620 14:57:46.995999 5963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:46 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:46.999593 5963 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:57:47.001338 5963 webserver.cc:460] Webserver started at http://127.21.159.194:44175/ using document root <none> and password file <none>
I20260620 14:57:47.002032 5963 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:47.002158 5963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:47.008468 5963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:57:47.011631 5978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:47.013355 5963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:57:47.013518 5963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "e4cb01e03e3f479d97a242f00b4ca9e3"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "011ff8d143ceb974c83eb3b52413211f"
server_key_iv: "08cfe11ecbbd771819a251a7254d9dc6"
server_key_version: "encryptionkey@0"
I20260620 14:57:47.014570 5963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:47.028710 5963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:47.033303 5963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:47.033586 5963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:47.034623 5963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:47.036041 5963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:47.036142 5963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:47.036216 5963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:47.036264 5963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:47.073843 5963 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:40283
I20260620 14:57:47.073899 6090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:40283 every 8 connection(s)
I20260620 14:57:47.075521 5963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:57:47.084388 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 5963
I20260620 14:57:47.085049 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:34179
--local_ip_for_outbound_sockets=127.21.159.195
--tserver_master_addrs=127.21.159.254:42219
--webserver_port=36439
--webserver_interface=127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:57:47 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967466, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:47.100886 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.078608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:44161 (local address 127.21.159.254:42219)
0620 14:57:47.078873 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.078881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.080238 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 14:57:47.083455 (+ 3217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.083460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.083572 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.083670 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:47.086004 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.087223 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.089102 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.089911 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.090061 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.094501 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:47.094507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:47.094510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:47.094548 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:47.097324 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.098345 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.098349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.098351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.098471 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.098979 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.098982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.098984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.099352 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:47.099468 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.100412 (+ 944us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.100614 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:57:47.102535 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.079140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:44161)
0620 14:57:47.080050 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:47.080104 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:47.082982 (+ 2878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:47.084049 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:47.084063 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:47.084968 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:47.085766 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.085790 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.087420 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.087424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.088920 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.088936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.090236 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.090241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.090466 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.091326 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:47.091351 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:47.094224 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:47.097502 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:47.097515 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:47.097539 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:47.098217 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:47.098588 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:47.098596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:47.098603 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:47.098866 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:47.099495 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:47.099505 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:47.100001 (+ 496us) client_negotiation.cc:769] Sending connection context
0620 14:57:47.100328 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 14:57:47.100941 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":324,"threads_started":1}
I20260620 14:57:47.104311 6091 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42219
I20260620 14:57:47.104741 6091 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:47.105790 6091 heartbeater.cc:511] Master 127.21.159.254:42219 requested a full tablet report, sending...
I20260620 14:57:47.107925 5773 ts_manager.cc:194] Registered new tserver with Master: e4cb01e03e3f479d97a242f00b4ca9e3 (127.21.159.194:40283)
I20260620 14:57:47.108949 5773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:44161
W20260620 14:57:47.417212 6094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:57:47.417482 6094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:57:47.417575 6094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:57:47.425518 6094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:57:47.425634 6094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:57:47.425756 6094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:57:47.435851 6094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:34179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=36439
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:57:47.437135 6094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:57:47.438411 6094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:57:47.452083 6099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:47.452064 6102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:57:47.452138 6100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:57:47.452893 6094 server_base.cc:1034] running on GCE node
I20260620 14:57:47.453493 6094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:57:47.454443 6094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:57:47.455674 6094 hybrid_clock.cc:648] HybridClock initialized: now 1781967467455600 us; error 58 us; skew 500 ppm
I20260620 14:57:47.456076 6094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:57:47 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967467, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:57:47.459748 6094 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:57:47.461449 6094 webserver.cc:460] Webserver started at http://127.21.159.195:36439/ using document root <none> and password file <none>
I20260620 14:57:47.462211 6094 fs_manager.cc:359] Metadata directory not provided
I20260620 14:57:47.462363 6094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:57:47.468195 6094 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:57:47.470947 6109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:47.472555 6094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:57:47.472749 6094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "af37ae04112f4f4fa0e97f545aae4817"
format_stamp: "Formatted at 2026-06-20 14:57:24 on dist-test-slave-3mqx"
server_key: "49a50d393d8a370431f6fcfa13d52d43"
server_key_iv: "1f1624585790a5386ef0011c0256e025"
server_key_version: "encryptionkey@0"
I20260620 14:57:47.473899 6094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:57:47.496260 6094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:57:47.501292 6094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:57:47.501643 6094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:57:47.502876 6094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:57:47.504516 6094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:57:47.504624 6094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:47.504753 6094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:57:47.504835 6094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:57:47.542524 6094 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:34179
I20260620 14:57:47.542579 6221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:34179 every 8 connection(s)
I20260620 14:57:47.544284 6094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:57:47.547478 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 6094
Jun 20 14:57:47 dist-test-slave-3mqx krb5kdc[5189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967467, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:57:47.572162 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.548588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.21.159.254:42219)
0620 14:57:47.553630 (+ 5042us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.553634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.553648 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:57:47.553690 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.553693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.553741 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.553825 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:47.556237 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.557357 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.559821 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.560705 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.560874 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.564830 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:47.564840 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:47.564845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:47.564921 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:47.569022 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.569903 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.569907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.569909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.570034 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.570531 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.570534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.570536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.570936 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:47.571315 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.571583 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.571776 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":4949,"thread_start_us":4809,"threads_started":1}
I20260620 14:57:47.573829 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.547349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:57481 (local address 127.21.159.254:42219)
0620 14:57:47.547588 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.547596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.549393 (+ 1797us) server_negotiation.cc:407] Connection header received
0620 14:57:47.553192 (+ 3799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.553197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.553270 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.553418 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:47.555981 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.557357 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.560133 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.561010 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.561145 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.565990 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:47.565997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:47.566003 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:47.566061 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:47.569717 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.570802 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.570813 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.570815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.570956 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.571610 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.571615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.571618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.572001 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:47.572182 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.573200 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.573428 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":80}
I20260620 14:57:47.575035 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.548009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:57481)
0620 14:57:47.549168 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:47.549246 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:47.552755 (+ 3509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:47.553734 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:47.553743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:47.554713 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:57:47.555746 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.555778 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.557592 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.557598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.559921 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.559936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.561290 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.561300 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.561654 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.562440 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:57:47.562485 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:57:47.565683 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:57:47.569905 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:47.569914 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:47.569931 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:47.570647 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:47.571149 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:57:47.571154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:57:47.571158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:57:47.571459 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:57:47.572180 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:57:47.572190 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:57:47.572719 (+ 529us) client_negotiation.cc:769] Sending connection context
0620 14:57:47.573099 (+ 380us) client_negotiation.cc:240] Negotiation successful
0620 14:57:47.573727 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":417,"threads_started":1}
I20260620 14:57:47.576534 6222 heartbeater.cc:347] Connected to a master server at 127.21.159.254:42219
I20260620 14:57:47.576871 6222 heartbeater.cc:464] Registering TS with master...
I20260620 14:57:47.577759 6222 heartbeater.cc:511] Master 127.21.159.254:42219 requested a full tablet report, sending...
I20260620 14:57:47.579453 5773 ts_manager.cc:194] Registered new tserver with Master: af37ae04112f4f4fa0e97f545aae4817 (127.21.159.195:34179)
I20260620 14:57:47.580394 5773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:57481
I20260620 14:57:47.587874 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:57:47.595494 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.588939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.21.159.254:42219)
0620 14:57:47.589177 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.589186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.589595 (+ 409us) server_negotiation.cc:407] Connection header received
0620 14:57:47.589810 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.589814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.589966 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.590137 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:47.590553 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.591523 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.593306 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.593776 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.593890 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.594516 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:47.594755 (+ 239us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:57:47.595241 (+ 486us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33944: token expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 14:57:47.595672 5961 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33944: token expired
W20260620 14:57:47.596298 5696 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.159.254:42219)
I20260620 14:57:47.617738 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.596856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.21.159.254:42219)
0620 14:57:47.597028 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.597032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.597104 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:57:47.597260 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.597264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.597328 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.597409 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:47.598811 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.599782 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.601523 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.602003 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.602184 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.610983 (+ 8799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:47.610990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:47.610993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:47.611187 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:47.614042 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.615236 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.615241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.615244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.615401 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:47.615875 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:47.615880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:47.615883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:47.616345 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:47.616550 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.617076 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.617361 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:57:47.621790 5773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:57:47.629473 5773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:57:47.649124 5959 heartbeater.cc:503] Master 127.21.159.254:42219 was elected leader, sending a full tablet report...
I20260620 14:57:47.657424 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.642746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46726 (local address 127.21.159.193:40357)
0620 14:57:47.644852 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.644863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.644893 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:57:47.644996 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.645007 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.645279 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.646140 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:47.646409 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.650813 (+ 4404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.653852 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.655686 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.655912 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.656126 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.656886 (+ 760us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.657082 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1917,"thread_start_us":1743,"threads_started":1}
I20260620 14:57:47.657542 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.642561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:40357 (local address 127.0.0.1:46726)
0620 14:57:47.643638 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:47.643693 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:47.643879 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:47.645534 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:47.645541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:47.645598 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:47.646128 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.646149 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.651025 (+ 4876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.651036 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.653663 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.653679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.656290 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.656297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.656559 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.656636 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:57:47.656782 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:57:47.657023 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":43,"thread_start_us":568,"threads_started":1}
I20260620 14:57:47.658845 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.642664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:40283 (local address 127.0.0.1:48698)
0620 14:57:47.644545 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:47.644569 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:47.644703 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:47.645637 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:47.645642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:47.645676 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:47.646198 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.646215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.649137 (+ 2922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.649145 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.655698 (+ 6553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.655714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.657950 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.657956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.658154 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.658196 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:57:47.658310 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 14:57:47.658444 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1734,"thread_start_us":2139,"threads_started":1}
I20260620 14:57:47.659744 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.643188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48698 (local address 127.21.159.194:40283)
0620 14:57:47.644301 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.644313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.644751 (+ 438us) server_negotiation.cc:407] Connection header received
0620 14:57:47.644872 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.644881 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.645155 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.645362 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:47.646415 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.648919 (+ 2504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.655921 (+ 7002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.657624 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.657822 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.658022 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.659213 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.659400 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":759,"threads_started":1}
I20260620 14:57:47.662249 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.644983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:34179 (local address 127.0.0.1:38940)
0620 14:57:47.653447 (+ 8464us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:47.653464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:47.653594 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:47.654590 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:47.654597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:47.654663 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:47.655158 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.655173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.656769 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.656776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.659383 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.659399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.661339 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.661345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.661593 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.661641 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:57:47.661734 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:57:47.661870 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8334,"mutex_wait_us":1,"thread_start_us":8157,"threads_started":1}
I20260620 14:57:47.662319 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.645599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.21.159.195:34179)
0620 14:57:47.652896 (+ 7297us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.652910 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.653597 (+ 687us) server_negotiation.cc:407] Connection header received
0620 14:57:47.653753 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.653765 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.654048 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.654284 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:47.655372 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.656640 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.659584 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.661110 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.661339 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.661510 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.661802 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.661956 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7028,"thread_start_us":6827,"threads_started":1}
I20260620 14:57:47.662403 5894 tablet_service.cc:1467] Processing CreateTablet for tablet 6d615be74dd942bcbac0c0b101f137c3 (DEFAULT_TABLE table=test-table [id=723299d7c3b44f649eb13e795f35952b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:47.664172 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d615be74dd942bcbac0c0b101f137c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:47.663647 6026 tablet_service.cc:1467] Processing CreateTablet for tablet 6d615be74dd942bcbac0c0b101f137c3 (DEFAULT_TABLE table=test-table [id=723299d7c3b44f649eb13e795f35952b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:47.665552 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d615be74dd942bcbac0c0b101f137c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:47.665846 6157 tablet_service.cc:1467] Processing CreateTablet for tablet 6d615be74dd942bcbac0c0b101f137c3 (DEFAULT_TABLE table=test-table [id=723299d7c3b44f649eb13e795f35952b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:57:47.666957 6157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d615be74dd942bcbac0c0b101f137c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:57:47.679085 6238 tablet_bootstrap.cc:492] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Bootstrap starting.
I20260620 14:57:47.681738 6239 tablet_bootstrap.cc:492] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Bootstrap starting.
I20260620 14:57:47.682695 6238 tablet_bootstrap.cc:654] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:47.683699 6240 tablet_bootstrap.cc:492] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Bootstrap starting.
I20260620 14:57:47.684084 6238 log.cc:824] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:47.685400 6239 tablet_bootstrap.cc:654] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:47.686678 6239 log.cc:824] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:47.687389 6240 tablet_bootstrap.cc:654] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Neither blocks nor log segments found. Creating new log.
I20260620 14:57:47.687904 6238 tablet_bootstrap.cc:492] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: No bootstrap required, opened a new log
I20260620 14:57:47.688309 6238 ts_tablet_manager.cc:1397] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260620 14:57:47.688853 6240 log.cc:824] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:57:47.691009 6239 tablet_bootstrap.cc:492] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: No bootstrap required, opened a new log
I20260620 14:57:47.691368 6239 ts_tablet_manager.cc:1397] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260620 14:57:47.692215 6240 tablet_bootstrap.cc:492] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: No bootstrap required, opened a new log
I20260620 14:57:47.692488 6240 ts_tablet_manager.cc:1397] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 14:57:47.696113 6238 raft_consensus.cc:348] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.696516 6238 raft_consensus.cc:374] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:47.696687 6238 raft_consensus.cc:729] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af37ae04112f4f4fa0e97f545aae4817, State: Initialized, Role: FOLLOWER
I20260620 14:57:47.697333 6238 consensus_queue.cc:260] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.700300 6222 heartbeater.cc:503] Master 127.21.159.254:42219 was elected leader, sending a full tablet report...
I20260620 14:57:47.702431 6238 ts_tablet_manager.cc:1428] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260620 14:57:47.703012 6239 raft_consensus.cc:348] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.703553 6239 raft_consensus.cc:374] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:47.703704 6239 raft_consensus.cc:729] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603af19a73ea4daeb9dceb0d9827dabc, State: Initialized, Role: FOLLOWER
I20260620 14:57:47.703824 6240 raft_consensus.cc:348] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.704249 6240 raft_consensus.cc:374] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:57:47.704349 6240 raft_consensus.cc:729] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cb01e03e3f479d97a242f00b4ca9e3, State: Initialized, Role: FOLLOWER
I20260620 14:57:47.704399 6239 consensus_queue.cc:260] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.704968 6240 consensus_queue.cc:260] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.706632 6239 ts_tablet_manager.cc:1428] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260620 14:57:47.707819 6091 heartbeater.cc:503] Master 127.21.159.254:42219 was elected leader, sending a full tablet report...
I20260620 14:57:47.708505 6240 ts_tablet_manager.cc:1428] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
W20260620 14:57:47.796123 6223 tablet.cc:2369] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:47.827958 6092 tablet.cc:2369] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:57:47.862341 5960 tablet.cc:2369] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:57:47.895036 6246 raft_consensus.cc:482] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:57:47.895358 6246 raft_consensus.cc:504] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.896956 6246 leader_election.cc:288] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357), af37ae04112f4f4fa0e97f545aae4817 (127.21.159.195:34179)
I20260620 14:57:47.905007 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.897553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:42349 (local address 127.21.159.193:40357)
0620 14:57:47.897857 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.897867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.898469 (+ 602us) server_negotiation.cc:407] Connection header received
0620 14:57:47.898584 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.898589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.898669 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.898765 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:47.899592 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.900668 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.902719 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.903806 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.903932 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.903985 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.904611 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.904726 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:57:47.905431 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.897416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:40357 (local address 127.21.159.194:42349)
0620 14:57:47.898168 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:47.898191 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:47.898358 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:47.898913 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:47.898917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:47.898939 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:47.899396 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.899409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.900829 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.900835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.902536 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.902548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.904143 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.904152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.904364 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.904410 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:57:47.904550 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 14:57:47.904716 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":3,"thread_start_us":376,"threads_started":1}
I20260620 14:57:47.906164 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.898444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:58453 (local address 127.21.159.195:34179)
0620 14:57:47.898686 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.898693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.899222 (+ 529us) server_negotiation.cc:407] Connection header received
0620 14:57:47.899390 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.899394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.899477 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.899575 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:57:47.900307 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.901312 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.903463 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.904815 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.905012 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.905104 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.905711 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.905844 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 14:57:47.906735 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d615be74dd942bcbac0c0b101f137c3" candidate_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603af19a73ea4daeb9dceb0d9827dabc" is_pre_election: true
I20260620 14:57:47.906936 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.898206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:34179 (local address 127.21.159.194:58453)
0620 14:57:47.899101 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 14:57:47.899130 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 14:57:47.899252 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:57:47.899615 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:57:47.899619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:57:47.899648 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:57:47.900045 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.900058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.901470 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.901474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.903251 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:57:47.903265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.905189 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.905196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:57:47.905458 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.905486 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:57:47.906390 (+ 904us) client_negotiation.cc:240] Negotiation successful
0620 14:57:47.906557 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":22,"thread_start_us":525,"threads_started":1}
I20260620 14:57:47.907394 5914 raft_consensus.cc:2454] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cb01e03e3f479d97a242f00b4ca9e3 in term 0.
I20260620 14:57:47.908152 5982 leader_election.cc:302] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 603af19a73ea4daeb9dceb0d9827dabc, e4cb01e03e3f479d97a242f00b4ca9e3; no voters:
I20260620 14:57:47.908221 6177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d615be74dd942bcbac0c0b101f137c3" candidate_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af37ae04112f4f4fa0e97f545aae4817" is_pre_election: true
I20260620 14:57:47.908624 6246 raft_consensus.cc:2781] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:57:47.908813 6246 raft_consensus.cc:482] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:57:47.908953 6246 raft_consensus.cc:3037] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:47.908885 6177 raft_consensus.cc:2454] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4cb01e03e3f479d97a242f00b4ca9e3 in term 0.
I20260620 14:57:47.911778 6246 raft_consensus.cc:504] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.912431 6246 leader_election.cc:288] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [CANDIDATE]: Term 1 election: Requested vote from peers 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357), af37ae04112f4f4fa0e97f545aae4817 (127.21.159.195:34179)
I20260620 14:57:47.913054 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d615be74dd942bcbac0c0b101f137c3" candidate_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "603af19a73ea4daeb9dceb0d9827dabc"
I20260620 14:57:47.913134 6177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d615be74dd942bcbac0c0b101f137c3" candidate_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af37ae04112f4f4fa0e97f545aae4817"
I20260620 14:57:47.913303 5914 raft_consensus.cc:3037] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:47.913409 6177 raft_consensus.cc:3037] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:57:47.916006 6177 raft_consensus.cc:2454] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cb01e03e3f479d97a242f00b4ca9e3 in term 1.
I20260620 14:57:47.916069 5914 raft_consensus.cc:2454] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4cb01e03e3f479d97a242f00b4ca9e3 in term 1.
I20260620 14:57:47.916568 5982 leader_election.cc:302] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af37ae04112f4f4fa0e97f545aae4817, e4cb01e03e3f479d97a242f00b4ca9e3; no voters:
I20260620 14:57:47.916968 6246 raft_consensus.cc:2781] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:57:47.917548 6246 raft_consensus.cc:686] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 LEADER]: Becoming Leader. State: Replica: e4cb01e03e3f479d97a242f00b4ca9e3, State: Running, Role: LEADER
I20260620 14:57:47.918154 6246 consensus_queue.cc:237] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } }
I20260620 14:57:47.923799 5773 catalog_manager.cc:5495] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4cb01e03e3f479d97a242f00b4ca9e3 (127.21.159.194). New cstate: current_term: 1 leader_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4cb01e03e3f479d97a242f00b4ca9e3" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 40283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 } health_report { overall_health: UNKNOWN } } }
I20260620 14:57:47.960726 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:47.954335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.21.159.194:40283)
0620 14:57:47.954574 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:47.954579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:47.954599 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:57:47.954801 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:47.954805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:47.954888 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:47.955150 (+ 262us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:57:47.955480 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.956303 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.958074 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:47.958537 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:47.958658 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:47.959301 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:57:47.959944 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:57:47.960117 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:47.960248 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:57:47.960358 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 14:57:47.970564 6177 raft_consensus.cc:1261] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 1 FOLLOWER]: Refusing update from remote peer e4cb01e03e3f479d97a242f00b4ca9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:47.970559 5914 raft_consensus.cc:1261] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Refusing update from remote peer e4cb01e03e3f479d97a242f00b4ca9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:57:47.971745 6249 consensus_queue.cc:1035] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af37ae04112f4f4fa0e97f545aae4817" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 34179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:47.972204 6246 consensus_queue.cc:1035] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "603af19a73ea4daeb9dceb0d9827dabc" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:57:47.985428 6252 mvcc.cc:205] Tried to move back new op lower bound from 7298938748795277312 to 7298938748597891072. Current Snapshot: MvccSnapshot[applied={T|T < 7298938748795277312}]
I20260620 14:57:48.000658 5773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:48.001001 5773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:57:48.005491 5773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Sending DeleteTablet for 3 replicas of tablet 6d615be74dd942bcbac0c0b101f137c3
I20260620 14:57:48.007107 6157 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d615be74dd942bcbac0c0b101f137c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:47 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260620 14:57:48.007479 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5810
I20260620 14:57:48.007972 5894 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d615be74dd942bcbac0c0b101f137c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:47 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:46726
I20260620 14:57:48.008797 6026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d615be74dd942bcbac0c0b101f137c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:57:47 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:48698
I20260620 14:57:48.009006 6263 tablet_replica.cc:330] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: stopping tablet replica
I20260620 14:57:48.009505 6263 raft_consensus.cc:2229] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:48.010128 6263 raft_consensus.cc:2258] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:48.012454 6263 ts_tablet_manager.cc:1905] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:48.020828 6263 ts_tablet_manager.cc:1918] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:48.021114 6263 log.cc:1196] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/6d615be74dd942bcbac0c0b101f137c3
I20260620 14:57:48.021795 6263 ts_tablet_manager.cc:1939] T 6d615be74dd942bcbac0c0b101f137c3 P 603af19a73ea4daeb9dceb0d9827dabc: Deleting consensus metadata
I20260620 14:57:48.023641 6264 tablet_replica.cc:330] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: stopping tablet replica
I20260620 14:57:48.024216 5760 catalog_manager.cc:4841] TS 603af19a73ea4daeb9dceb0d9827dabc (127.21.159.193:40357): tablet 6d615be74dd942bcbac0c0b101f137c3 (table test-table [id=723299d7c3b44f649eb13e795f35952b]) successfully deleted
I20260620 14:57:48.024303 6264 raft_consensus.cc:2229] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:57:48.025231 6264 raft_consensus.cc:2258] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:48.026855 6262 tablet_replica.cc:330] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: stopping tablet replica
I20260620 14:57:48.027447 6262 raft_consensus.cc:2229] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:57:48.027858 6264 ts_tablet_manager.cc:1905] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:48.028201 6262 raft_consensus.cc:2258] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:57:48.029604 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.008352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.21.159.193:40357)
0620 14:57:48.009086 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.009092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.009111 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:57:48.009173 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.009178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.009251 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.009364 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.011896 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.013046 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.015616 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.016319 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.016467 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.020986 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.021003 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.021023 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.021075 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.025560 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.026503 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.026513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.026517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.026685 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.027225 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.027231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.027234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.027795 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.028207 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.028902 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.029162 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597}
I20260620 14:57:48.030375 5954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:48.032037 6262 ts_tablet_manager.cc:1905] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:57:48.036944 6264 ts_tablet_manager.cc:1918] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:48.037250 6264 log.cc:1196] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/6d615be74dd942bcbac0c0b101f137c3
I20260620 14:57:48.038076 6264 ts_tablet_manager.cc:1939] T 6d615be74dd942bcbac0c0b101f137c3 P e4cb01e03e3f479d97a242f00b4ca9e3: Deleting consensus metadata
I20260620 14:57:48.039971 6262 ts_tablet_manager.cc:1918] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:57:48.040199 6262 log.cc:1196] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/6d615be74dd942bcbac0c0b101f137c3
I20260620 14:57:48.040237 5759 catalog_manager.cc:4841] TS e4cb01e03e3f479d97a242f00b4ca9e3 (127.21.159.194:40283): tablet 6d615be74dd942bcbac0c0b101f137c3 (table test-table [id=723299d7c3b44f649eb13e795f35952b]) successfully deleted
I20260620 14:57:48.040858 6262 ts_tablet_manager.cc:1939] T 6d615be74dd942bcbac0c0b101f137c3 P af37ae04112f4f4fa0e97f545aae4817: Deleting consensus metadata
I20260620 14:57:48.043488 5760 catalog_manager.cc:4841] TS af37ae04112f4f4fa0e97f545aae4817 (127.21.159.195:34179): tablet 6d615be74dd942bcbac0c0b101f137c3 (table test-table [id=723299d7c3b44f649eb13e795f35952b]) successfully deleted
I20260620 14:57:48.147058 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5810
I20260620 14:57:48.166069 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5963
I20260620 14:57:48.181044 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.166945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.21.159.194:40283)
0620 14:57:48.167175 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.167182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.167269 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:57:48.167484 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.167490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.167607 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.167719 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.169049 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.169873 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.171181 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.171665 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.171805 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.174520 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.174543 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.174567 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.174604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.178082 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.178816 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.178827 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.178830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.178977 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.179404 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.179410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.179413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.179791 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.180144 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.180452 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.180685 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:57:48.181807 6086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:48.296484 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5963
I20260620 14:57:48.317885 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 6094
I20260620 14:57:48.334404 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:57:48.318763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.21.159.195:34179)
0620 14:57:48.318936 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:57:48.318940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:57:48.318958 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:57:48.319341 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:57:48.319346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:57:48.319421 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:57:48.319517 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:57:48.321283 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.322089 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.323845 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:57:48.324264 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:57:48.324365 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:57:48.327264 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:57:48.327281 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:57:48.327297 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:57:48.327354 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:57:48.330856 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.332002 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.332025 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.332028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.332197 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:57:48.332709 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:57:48.332713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:57:48.332716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:57:48.333203 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:57:48.333476 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 14:57:48.333820 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:57:48.334111 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:57:48.335278 6217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:48.431255 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 6094
I20260620 14:57:48.448065 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 5740
I20260620 14:57:48.449381 5803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:57:48.564899 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 5740
I20260620 14:58:08.587936 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42219
--webserver_interface=127.21.159.254
--webserver_port=33601
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:08.946662 6269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:08.946971 6269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:08.947167 6269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:08.954622 6269 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:58:08.954730 6269 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:58:08.954802 6269 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:58:08.954877 6269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:08.954969 6269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:08.955112 6269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:08.955199 6269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:08.955276 6269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:08.965442 6269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=33601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:08.966838 6269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:08.968281 6269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:08.981024 6274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:08.981001 6275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:08.981001 6277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:08.981521 6269 server_base.cc:1034] running on GCE node
I20260620 14:58:08.982560 6269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:08.984047 6269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:08.985280 6269 hybrid_clock.cc:648] HybridClock initialized: now 1781967488985226 us; error 26 us; skew 500 ppm
I20260620 14:58:08.985574 6269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:08 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967488, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:08.992798 6269 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:58:08.994593 6269 webserver.cc:460] Webserver started at http://127.21.159.254:33601/ using document root <none> and password file <none>
I20260620 14:58:08.995455 6269 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:08.995556 6269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:09.001682 6269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:58:09.005345 6284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.006901 6269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:58:09.007220 6269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "26d4dea6f4974c75b5551530f43cbc29"
format_stamp: "Formatted at 2026-06-20 14:57:22 on dist-test-slave-3mqx"
server_key: "bf27f655a82ffc42d93dc70d6eaf6ff8"
server_key_iv: "cfd3860ee205fb59d915ad548d24ef52"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.008329 6269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:09.025808 6269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:09.030705 6269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:09.031008 6269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:09.051271 6269 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:42219
I20260620 14:58:09.051293 6335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:42219 every 8 connection(s)
I20260620 14:58:09.052891 6269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:09.054385 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 6269
I20260620 14:58:09.055598 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:40357
--local_ip_for_outbound_sockets=127.21.159.193
--tserver_master_addrs=127.21.159.254:42219
--webserver_port=37337
--webserver_interface=127.21.159.193
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:58:09.063521 6336 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:58:09.067396 6336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Bootstrap starting.
I20260620 14:58:09.089720 6336 log.cc:824] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:09.105866 6336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:58:09.106472 6336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Bootstrap complete.
I20260620 14:58:09.116209 6336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:58:09.117158 6336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d4dea6f4974c75b5551530f43cbc29, State: Initialized, Role: FOLLOWER
I20260620 14:58:09.117799 6336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:58:09.118049 6336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:09.118192 6336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:09.118368 6336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:58:09.122460 6336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:58:09.122994 6336 leader_election.cc:302] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d4dea6f4974c75b5551530f43cbc29; no voters:
I20260620 14:58:09.123760 6336 leader_election.cc:288] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:58:09.123910 6340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:58:09.125022 6340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [term 3 LEADER]: Becoming Leader. State: Replica: 26d4dea6f4974c75b5551530f43cbc29, State: Running, Role: LEADER
I20260620 14:58:09.125686 6340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } }
I20260620 14:58:09.126006 6336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:09.128247 6342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d4dea6f4974c75b5551530f43cbc29. Latest consensus state: current_term: 3 leader_uuid: "26d4dea6f4974c75b5551530f43cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } } }
I20260620 14:58:09.128391 6341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "26d4dea6f4974c75b5551530f43cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d4dea6f4974c75b5551530f43cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 42219 } } }
I20260620 14:58:09.128695 6342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:09.128806 6341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:09.138362 6347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:09.149398 6347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:09.151517 6347 catalog_manager.cc:1259] Loaded cluster ID: 04b3aced9d1d4e44a4d9a7e9c6317f6a
I20260620 14:58:09.151718 6347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:09.157521 6347 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:09.159849 6347 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Loaded TSK: 0
I20260620 14:58:09.160095 6347 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 26d4dea6f4974c75b5551530f43cbc29: Loaded TSK: 1
F20260620 14:58:09.161651 6347 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fda246449ad google::LogMessage::Fail() at ??:0
@ 0x7fda2464788a google::LogMessage::SendToLog() at ??:0
@ 0x7fda246444de google::LogMessage::Flush() at ??:0
@ 0x7fda24645a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fda283a87d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fda2839773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fda3a0731c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fda3a072783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fda3a0fe2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fda3a0fe0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fda3a1485e9 std::function<>::operator()() at ??:0
@ 0x7fda3a076cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fda3a074c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fda3a0fc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fda3a0fc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fda27290345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fda2729391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fda27293799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fda2726eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fda2a5486db start_thread at ??:0
@ 0x7fda234de61f clone at ??:0
W20260620 14:58:09.477149 6338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:09.477463 6338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:09.477561 6338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:09.485224 6338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:09.485348 6338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:09.485471 6338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:58:09.496018 6338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:40357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=37337
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:09.497354 6338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:09.498662 6338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:09.512830 6386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.512887 6384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.513051 6383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.514385 6338 server_base.cc:1034] running on GCE node
I20260620 14:58:09.515339 6338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:09.516513 6338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:09.517809 6338 hybrid_clock.cc:648] HybridClock initialized: now 1781967489517747 us; error 46 us; skew 500 ppm
I20260620 14:58:09.518217 6338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:09 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:09.521909 6338 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:58:09.523692 6338 webserver.cc:460] Webserver started at http://127.21.159.193:37337/ using document root <none> and password file <none>
I20260620 14:58:09.524411 6338 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:09.524523 6338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:09.530587 6338 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:58:09.533886 6393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.535777 6338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:58:09.535944 6338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "603af19a73ea4daeb9dceb0d9827dabc"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "6cca551ad4d32bc8e46dcd14f1a6dbfd"
server_key_iv: "8c06c215b6ca53a09bd9feb0e19de8e7"
server_key_version: "encryptionkey@0"
I20260620 14:58:09.537000 6338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:09.558365 6338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:09.562922 6338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:09.563258 6338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:09.564441 6338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:09.565905 6338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:09.566005 6338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.566084 6338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:09.566151 6338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:09.601840 6338 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:40357
I20260620 14:58:09.601948 6505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:40357 every 8 connection(s)
I20260620 14:58:09.603493 6338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:09.605206 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 6338
I20260620 14:58:09.606061 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:40283
--local_ip_for_outbound_sockets=127.21.159.194
--tserver_master_addrs=127.21.159.254:42219
--webserver_port=44175
--webserver_interface=127.21.159.194
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:58:09.609782 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.607365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:46315)
0620 14:58:09.608332 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.608742 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":378,"threads_started":1}
W20260620 14:58:09.610601 6506 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:42219 (0 consecutive failures): Network error: Failed to ping master at 127.21.159.254:42219: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
I20260620 14:58:09.611706 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.611302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:36539)
0620 14:58:09.611483 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.611579 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:58:09.612671 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:09.612297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:60925)
0620 14:58:09.612448 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:09.612549 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:58:09.612936 6506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:58:09.951300 6509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:09.951663 6509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:09.951840 6509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:09.959354 6509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:09.959535 6509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:09.959733 6509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:58:09.969858 6509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:40283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=44175
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:09.971294 6509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:09.972752 6509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:09.986176 6514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.986277 6517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:09.986210 6515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:09.987447 6509 server_base.cc:1034] running on GCE node
I20260620 14:58:09.988114 6509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:09.989159 6509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:09.990417 6509 hybrid_clock.cc:648] HybridClock initialized: now 1781967489990358 us; error 47 us; skew 500 ppm
I20260620 14:58:09.990842 6509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:09 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967489, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:09.994338 6509 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:58:09.996068 6509 webserver.cc:460] Webserver started at http://127.21.159.194:44175/ using document root <none> and password file <none>
I20260620 14:58:09.996812 6509 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:09.996961 6509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:10.002944 6509 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:58:10.006095 6524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:10.007871 6509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:10.008070 6509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "e4cb01e03e3f479d97a242f00b4ca9e3"
format_stamp: "Formatted at 2026-06-20 14:57:23 on dist-test-slave-3mqx"
server_key: "011ff8d143ceb974c83eb3b52413211f"
server_key_iv: "08cfe11ecbbd771819a251a7254d9dc6"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.009150 6509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:10.046015 6509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:10.050817 6509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:10.051201 6509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:10.052510 6509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:10.054533 6509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:10.054708 6509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 14:58:10.054863 6509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:10.054962 6509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:10.093446 6509 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:40283
I20260620 14:58:10.093535 6636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:40283 every 8 connection(s)
I20260620 14:58:10.095000 6509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:10.100297 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.098183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:58459)
0620 14:58:10.098953 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.099290 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":288,"threads_started":1}
W20260620 14:58:10.101009 6637 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:42219 (0 consecutive failures): Network error: Failed to ping master at 127.21.159.254:42219: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
I20260620 14:58:10.101858 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.101483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:48049)
0620 14:58:10.101670 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.101745 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 14:58:10.102873 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.102525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:38229)
0620 14:58:10.102686 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.102762 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:58:10.103212 6637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:58:10.103538 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 6509
I20260620 14:58:10.104146 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:34179
--local_ip_for_outbound_sockets=127.21.159.195
--tserver_master_addrs=127.21.159.254:42219
--webserver_port=36439
--webserver_interface=127.21.159.195
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:10.457990 6640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:10.458316 6640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:10.458421 6640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:10.465958 6640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:10.466105 6640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:10.466231 6640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:58:10.476478 6640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:34179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=36439
--tserver_master_addrs=127.21.159.254:42219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:10.477933 6640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:10.479447 6640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:10.494215 6646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:10.494233 6648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:10.494270 6645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:10.495281 6640 server_base.cc:1034] running on GCE node
I20260620 14:58:10.496025 6640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:10.497126 6640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:10.498389 6640 hybrid_clock.cc:648] HybridClock initialized: now 1781967490498326 us; error 58 us; skew 500 ppm
I20260620 14:58:10.498790 6640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:10 dist-test-slave-3mqx krb5kdc[5189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967490, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:10.502354 6640 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:58:10.504076 6640 webserver.cc:460] Webserver started at http://127.21.159.195:36439/ using document root <none> and password file <none>
I20260620 14:58:10.504832 6640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:10.504948 6640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:10.511267 6640 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:58:10.514540 6655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:10.516191 6640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:10.516367 6640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "af37ae04112f4f4fa0e97f545aae4817"
format_stamp: "Formatted at 2026-06-20 14:57:24 on dist-test-slave-3mqx"
server_key: "49a50d393d8a370431f6fcfa13d52d43"
server_key_iv: "1f1624585790a5386ef0011c0256e025"
server_key_version: "encryptionkey@0"
I20260620 14:58:10.517405 6640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:10.533906 6640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:10.538581 6640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:10.538889 6640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:10.539960 6640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:10.541438 6640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:10.541532 6640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:10.541673 6640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:10.541744 6640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:10.575439 6640 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:34179
I20260620 14:58:10.575522 6767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:34179 every 8 connection(s)
I20260620 14:58:10.577252 6640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:10.580530 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 6640
I20260620 14:58:10.582718 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.580478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:57921)
0620 14:58:10.581373 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.581681 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":399,"threads_started":1}
W20260620 14:58:10.583369 6768 heartbeater.cc:644] Failed to heartbeat to 127.21.159.254:42219 (0 consecutive failures): Network error: Failed to ping master at 127.21.159.254:42219: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
I20260620 14:58:10.584203 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.583871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:40745)
0620 14:58:10.584025 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.584097 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:58:10.585026 6770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.584713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:34927)
0620 14:58:10.584844 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.584923 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 14:58:10.585270 6768 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:58:10.614547 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:10.613588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:42327)
0620 14:58:10.614122 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:10.614251 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":231,"threads_started":1}
I20260620 14:58:11.104496 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:11.103735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:38989)
0620 14:58:11.104185 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:11.104270 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":216,"threads_started":1}
I20260620 14:58:11.586817 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:11.585869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:39709)
0620 14:58:11.586400 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:11.586523 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":278,"threads_started":1}
I20260620 14:58:11.616309 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:11.615536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:42127)
0620 14:58:11.616030 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:11.616127 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":238,"threads_started":1}
I20260620 14:58:12.106309 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.105553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:47931)
0620 14:58:12.106028 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:12.106122 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260620 14:58:12.589073 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.588075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:53779)
0620 14:58:12.588753 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:12.588872 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":341,"threads_started":1}
I20260620 14:58:12.618131 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:12.617295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:44505)
0620 14:58:12.617827 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:12.617921 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":304,"threads_started":1}
I20260620 14:58:13.108094 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:13.107299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:42185)
0620 14:58:13.107816 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:13.107913 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":298,"threads_started":1}
I20260620 14:58:13.590891 6780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:13.590070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:56117)
0620 14:58:13.590610 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:13.590706 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260620 14:58:13.619831 6781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:13.619037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:60327)
0620 14:58:13.619564 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:13.619656 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":247,"threads_started":1}
I20260620 14:58:14.109764 6782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.108982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:44259)
0620 14:58:14.109447 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:14.109533 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
I20260620 14:58:14.593225 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.592221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:46677)
0620 14:58:14.592912 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:14.593016 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":355,"threads_started":1}
I20260620 14:58:14.621893 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:14.621033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:36211)
0620 14:58:14.621552 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:14.621652 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":269,"threads_started":1}
I20260620 14:58:15.111656 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.110781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:36143)
0620 14:58:15.111379 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.111467 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":357,"threads_started":1}
I20260620 14:58:15.595407 6786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.594528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:54033)
0620 14:58:15.595149 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.595234 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":300,"threads_started":1}
I20260620 14:58:15.623781 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:15.622955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:56357)
0620 14:58:15.623496 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:15.623607 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":312,"threads_started":1}
I20260620 14:58:16.113873 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.112830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:49223)
0620 14:58:16.113508 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.113626 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":332,"threads_started":1}
I20260620 14:58:16.597146 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.596428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:57793)
0620 14:58:16.596885 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.596985 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260620 14:58:16.625765 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:16.624824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:39909)
0620 14:58:16.625425 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:16.625523 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":304,"threads_started":1}
I20260620 14:58:17.116101 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:17.115008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:60203)
0620 14:58:17.115689 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:17.115829 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":272,"threads_started":1}
I20260620 14:58:17.599116 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:17.598199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:58255)
0620 14:58:17.598765 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:17.598856 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":285,"threads_started":1}
I20260620 14:58:17.627764 6793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:17.626874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:40557)
0620 14:58:17.627491 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:17.627587 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":326,"threads_started":1}
I20260620 14:58:18.118767 6794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.117673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:36753)
0620 14:58:18.118416 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.118552 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":405,"threads_started":1}
I20260620 14:58:18.600809 6795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.600055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:49873)
0620 14:58:18.600550 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.600639 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
I20260620 14:58:18.629637 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:18.628704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:40777)
0620 14:58:18.629311 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:18.629432 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":345,"threads_started":1}
I20260620 14:58:19.121090 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.119848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:59423)
0620 14:58:19.120490 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:19.120627 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":298,"threads_started":1}
I20260620 14:58:19.602708 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.601871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:55583)
0620 14:58:19.602375 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:19.602537 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":234,"threads_started":1}
I20260620 14:58:19.631433 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:19.630665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:37999)
0620 14:58:19.631178 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:19.631268 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260620 14:58:20.123312 6800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.122425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:58409)
0620 14:58:20.122946 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.123041 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":282,"threads_started":1}
I20260620 14:58:20.604682 6801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.603821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:45775)
0620 14:58:20.604420 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.604524 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":367,"threads_started":1}
I20260620 14:58:20.633287 6802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:20.632472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:39081)
0620 14:58:20.632994 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:20.633098 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":259,"threads_started":1}
I20260620 14:58:21.125144 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.124356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:56577)
0620 14:58:21.124851 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.124954 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":248,"threads_started":1}
I20260620 14:58:21.606531 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.605704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:53453)
0620 14:58:21.606238 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.606339 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":288,"threads_started":1}
I20260620 14:58:21.635439 6805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:21.634559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:57657)
0620 14:58:21.635134 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:21.635247 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":288,"threads_started":1}
I20260620 14:58:22.126905 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:22.126176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:53111)
0620 14:58:22.126635 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:22.126728 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":232,"threads_started":1}
I20260620 14:58:22.608494 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:22.607573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:54881)
0620 14:58:22.608184 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:22.608298 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":278,"threads_started":1}
I20260620 14:58:22.637379 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:22.636489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:47967)
0620 14:58:22.637015 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:22.637116 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":257,"threads_started":1}
I20260620 14:58:23.128787 6809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:23.127951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:54801)
0620 14:58:23.128474 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:23.128577 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":264,"threads_started":1}
I20260620 14:58:23.610461 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:23.609603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:49033)
0620 14:58:23.610137 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:23.610244 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":262,"threads_started":1}
I20260620 14:58:23.639495 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:23.638566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:39477)
0620 14:58:23.639159 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:23.639267 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":290,"threads_started":1}
I20260620 14:58:24.130921 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.130013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:56781)
0620 14:58:24.130620 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:24.130717 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":385,"threads_started":1}
I20260620 14:58:24.612273 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.611556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:40455)
0620 14:58:24.611999 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:24.612095 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":214,"threads_started":1}
I20260620 14:58:24.641698 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:24.640794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.193:56111)
0620 14:58:24.641371 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:24.641497 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
I20260620 14:58:25.132891 6815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.132114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.194:55583)
0620 14:58:25.132605 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.132702 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260620 14:58:25.586133 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 6338
I20260620 14:58:25.606586 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.587442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.21.159.193:40357)
0620 14:58:25.588238 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.588254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.588330 (+ 76us) server_negotiation.cc:407] Connection header received
0620 14:58:25.590075 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.590087 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.590565 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.590953 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.592532 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.593831 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.595282 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.596040 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.596194 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.599110 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:25.599128 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:25.599148 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:25.599202 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:25.602574 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.603505 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.603516 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.603526 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.603768 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.604153 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.604158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.604161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.604937 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:25.605286 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:25.605687 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 14:58:25.606177 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":443,"threads_started":1}
I20260620 14:58:25.608256 6501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:25.614260 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.613404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:42219 (local address 127.21.159.195:54963)
0620 14:58:25.613905 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:25.614033 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.254:42219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":272,"threads_started":1}
I20260620 14:58:25.694512 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 6338
I20260620 14:58:25.712826 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 6509
I20260620 14:58:25.730645 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.713925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.21.159.194:40283)
0620 14:58:25.714535 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.714549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.714587 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:58:25.716265 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.716276 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.716723 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.717031 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.718397 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.719812 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.721233 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.721923 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.722131 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.724834 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:25.724841 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:25.724857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:25.724912 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:25.727759 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.728456 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.728462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.728469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.728629 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.728947 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.728950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.728952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.729444 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:25.729673 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:25.730030 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:58:25.730366 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":327,"threads_started":1}
I20260620 14:58:25.732095 6632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:25.819628 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 6509
I20260620 14:58:25.837393 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 6640
I20260620 14:58:25.856242 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:25.838549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.21.159.195:34179)
0620 14:58:25.839374 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:25.839388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:25.839431 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:58:25.841130 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:25.841139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:25.841618 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:25.841925 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:25.843387 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.844759 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.846138 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:25.846897 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:25.847114 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:25.849690 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:25.849700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:25.849717 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:25.849756 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:25.853036 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.853757 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.853764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.853772 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.853943 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:25.854268 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:25.854272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:25.854274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:25.854823 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:25.855140 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:25.855498 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:58:25.855933 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":372,"threads_started":1}
I20260620 14:58:25.857669 6763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:25.944063 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 6640
2026-06-20T14:58:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65150 ms)
[----------] 30 tests from SecurityITest (305768 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:26 dist-test-slave-3mqx krb5kdc[6826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:26 dist-test-slave-3mqx krb5kdc[6826](info): set up 2 sockets
Jun 20 14:58:26 dist-test-slave-3mqx krb5kdc[6826](info): commencing operation
krb5kdc: starting...
W20260620 14:58:27.103257 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:27 dist-test-slave-3mqx krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:58:27.363847 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:32769
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:32769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:27.709864 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:27.710253 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:27.710412 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:27.717744 6839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:27.717929 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:27.718036 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:27.718142 6839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:27.718225 6839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:27.728842 6839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:32769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:32769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:27.730312 6839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:27.731732 6839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:27.744493 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:27.744534 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:27.744534 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:27.745301 6839 server_base.cc:1034] running on GCE node
I20260620 14:58:27.746178 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:27.747773 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:27.749068 6839 hybrid_clock.cc:648] HybridClock initialized: now 1781967507749028 us; error 53 us; skew 500 ppm
I20260620 14:58:27.749508 6839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:27 dist-test-slave-3mqx krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967507, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:27.753043 6839 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:58:27.755122 6839 webserver.cc:460] Webserver started at http://127.21.159.254:45157/ using document root <none> and password file <none>
I20260620 14:58:27.755898 6839 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:27.756053 6839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:27.756457 6839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:27.760037 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "51c7c2a6b13742378eb24a63cfda26d2"
format_stamp: "Formatted at 2026-06-20 14:58:27 on dist-test-slave-3mqx"
server_key: "f113e7a1f4470801743791e581151c20"
server_key_iv: "a60d17e133827c096ba213cff29833cf"
server_key_version: "encryptionkey@0"
I20260620 14:58:27.760846 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "51c7c2a6b13742378eb24a63cfda26d2"
format_stamp: "Formatted at 2026-06-20 14:58:27 on dist-test-slave-3mqx"
server_key: "f113e7a1f4470801743791e581151c20"
server_key_iv: "a60d17e133827c096ba213cff29833cf"
server_key_version: "encryptionkey@0"
I20260620 14:58:27.766582 6839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:27.770478 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:27.772099 6839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:27.772382 6839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "51c7c2a6b13742378eb24a63cfda26d2"
format_stamp: "Formatted at 2026-06-20 14:58:27 on dist-test-slave-3mqx"
server_key: "f113e7a1f4470801743791e581151c20"
server_key_iv: "a60d17e133827c096ba213cff29833cf"
server_key_version: "encryptionkey@0"
I20260620 14:58:27.772612 6839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:27.787420 6839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:27.796749 6839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:27.797101 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:27.817317 6839 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:32769
I20260620 14:58:27.817350 6905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:32769 every 8 connection(s)
I20260620 14:58:27.818773 6839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:27.819278 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 6839
I20260620 14:58:27.819515 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:27.820425 22143 external_mini_cluster.cc:1428] Setting key db39cd8bde6d222b5e1dbbcfab3f360a
I20260620 14:58:27.823984 6906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:27.835671 6906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2: Bootstrap starting.
Jun 20 14:58:27 dist-test-slave-3mqx krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:27.839382 6906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:27.840574 6906 log.cc:824] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:27.843628 6906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2: No bootstrap required, opened a new log
I20260620 14:58:27.845064 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:27.822706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.21.159.254:32769)
0620 14:58:27.823624 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:27.823637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:27.823680 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:58:27.825475 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:27.825499 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:27.825889 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:27.826319 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:27.828149 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.829666 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.831285 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:27.832049 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:27.832308 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:27.837289 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:27.837302 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:27.837319 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:27.837370 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:27.840890 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:27.841670 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:27.841681 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:27.841687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:27.841859 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:27.842287 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:27.842291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:27.842293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:27.842826 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:27.843146 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:27.843599 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 14:58:27.844058 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":425,"threads_started":1}
I20260620 14:58:27.852104 6906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c7c2a6b13742378eb24a63cfda26d2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32769 } }
I20260620 14:58:27.852558 6906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:27.852674 6906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c7c2a6b13742378eb24a63cfda26d2, State: Initialized, Role: FOLLOWER
I20260620 14:58:27.853317 6906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c7c2a6b13742378eb24a63cfda26d2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32769 } }
I20260620 14:58:27.853577 6906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:27.853729 6906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:27.853917 6906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:27.856640 6906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c7c2a6b13742378eb24a63cfda26d2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32769 } }
I20260620 14:58:27.857156 6906 leader_election.cc:302] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51c7c2a6b13742378eb24a63cfda26d2; no voters:
I20260620 14:58:27.857852 6906 leader_election.cc:288] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:27.858027 6911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:27.858855 6911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [term 1 LEADER]: Becoming Leader. State: Replica: 51c7c2a6b13742378eb24a63cfda26d2, State: Running, Role: LEADER
I20260620 14:58:27.859515 6911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c7c2a6b13742378eb24a63cfda26d2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32769 } }
I20260620 14:58:27.860191 6906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:27.862679 6913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51c7c2a6b13742378eb24a63cfda26d2. Latest consensus state: current_term: 1 leader_uuid: "51c7c2a6b13742378eb24a63cfda26d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c7c2a6b13742378eb24a63cfda26d2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32769 } } }
I20260620 14:58:27.863101 6913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:27.863395 6912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51c7c2a6b13742378eb24a63cfda26d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c7c2a6b13742378eb24a63cfda26d2" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32769 } } }
I20260620 14:58:27.863688 6912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:27.865159 6920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:27.871419 6920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:27.881038 6920 catalog_manager.cc:1347] Generated new cluster ID: 45bb7309958649ef820bb45216f76ab2
I20260620 14:58:27.881186 6920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:27.906997 6920 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:27.909108 6920 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:27.918531 6920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2: Generated new TSK 0
I20260620 14:58:27.919651 6920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:58:27.991403 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32769
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:28.333535 6932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:28.333820 6932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:28.333992 6932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:28.342262 6932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:28.342383 6932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:28.342500 6932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:58:28.352682 6932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:28.353962 6932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:28.355409 6932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:28.370219 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:28.370141 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:28.370167 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:28.376457 6932 server_base.cc:1034] running on GCE node
I20260620 14:58:28.377328 6932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:28.378443 6932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:28.379715 6932 hybrid_clock.cc:648] HybridClock initialized: now 1781967508379644 us; error 77 us; skew 500 ppm
I20260620 14:58:28.380110 6932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:28 dist-test-slave-3mqx krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:28.383740 6932 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:58:28.385494 6932 webserver.cc:460] Webserver started at http://127.21.159.193:36045/ using document root <none> and password file <none>
I20260620 14:58:28.386276 6932 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:28.386425 6932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:28.386757 6932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:28.389865 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "a781dcf039d148db81152aa9534c4b02"
format_stamp: "Formatted at 2026-06-20 14:58:28 on dist-test-slave-3mqx"
server_key: "32b98480cc94ba7a744086d1089d65cb"
server_key_iv: "7cf3f8f2b594b525dfe438b9a8e93863"
server_key_version: "encryptionkey@0"
I20260620 14:58:28.390663 6932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "a781dcf039d148db81152aa9534c4b02"
format_stamp: "Formatted at 2026-06-20 14:58:28 on dist-test-slave-3mqx"
server_key: "32b98480cc94ba7a744086d1089d65cb"
server_key_iv: "7cf3f8f2b594b525dfe438b9a8e93863"
server_key_version: "encryptionkey@0"
I20260620 14:58:28.400405 6932 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.000s
I20260620 14:58:28.406572 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:28.409031 6932 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 14:58:28.409297 6932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "a781dcf039d148db81152aa9534c4b02"
format_stamp: "Formatted at 2026-06-20 14:58:28 on dist-test-slave-3mqx"
server_key: "32b98480cc94ba7a744086d1089d65cb"
server_key_iv: "7cf3f8f2b594b525dfe438b9a8e93863"
server_key_version: "encryptionkey@0"
I20260620 14:58:28.409631 6932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:28.444670 6932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:28.449510 6932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:28.449774 6932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:28.450765 6932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:28.452243 6932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:28.452317 6932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:28.452402 6932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:28.452453 6932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:28.488883 6932 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:43989
I20260620 14:58:28.488976 7059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:43989 every 8 connection(s)
I20260620 14:58:28.490245 6932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:28.497871 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 6932
I20260620 14:58:28.498136 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:28.499118 22143 external_mini_cluster.cc:1428] Setting key 1893aeaae6be90505e6aacfb22b74fe1
Jun 20 14:58:28 dist-test-slave-3mqx krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:28.514063 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:28.493152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:35621 (local address 127.21.159.254:32769)
0620 14:58:28.493634 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:28.493639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:28.494683 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 14:58:28.496633 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:28.496638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:28.496961 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:28.497070 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:28.499843 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.500665 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.502322 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.502725 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.502862 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.507284 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:28.507291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:28.507294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:28.507333 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:28.510526 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:28.511623 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:28.511627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:28.511630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:28.511763 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:28.512221 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:28.512228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:28.512235 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:28.512501 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:28.512666 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:28.513429 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 14:58:28.513671 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":250,"threads_started":1}
I20260620 14:58:28.515170 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:28.493789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32769 (local address 127.21.159.193:35621)
0620 14:58:28.494496 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:28.494542 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:28.496329 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:28.497552 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:28.497567 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:28.498520 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:28.499611 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.499647 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.500855 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.500860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.502167 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:28.502179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:28.502974 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:28.502981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:28.503295 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:28.504142 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:28.504170 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:28.506901 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:28.510731 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:28.510741 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:28.510767 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:28.511425 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:28.511883 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:28.511888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:28.511893 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:28.512095 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:28.512638 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:28.512647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:28.513025 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 14:58:28.513307 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 14:58:28.513871 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":286,"threads_started":1}
I20260620 14:58:28.517059 7060 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32769
I20260620 14:58:28.517454 7060 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:28.518548 7060 heartbeater.cc:511] Master 127.21.159.254:32769 requested a full tablet report, sending...
I20260620 14:58:28.521663 6871 ts_manager.cc:194] Registered new tserver with Master: a781dcf039d148db81152aa9534c4b02 (127.21.159.193:43989)
I20260620 14:58:28.524314 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:35621
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:58:28.564227 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32769
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:28.901077 7066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:28.901489 7066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:28.901674 7066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:28.909277 7066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:28.909456 7066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:28.909631 7066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:58:28.920300 7066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:28.921651 7066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:28.922972 7066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:28.936686 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:28.936693 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:28.937088 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:28.937667 7066 server_base.cc:1034] running on GCE node
I20260620 14:58:28.938383 7066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:28.939440 7066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:28.940690 7066 hybrid_clock.cc:648] HybridClock initialized: now 1781967508940622 us; error 55 us; skew 500 ppm
I20260620 14:58:28.941015 7066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:28 dist-test-slave-3mqx krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:28.944233 7066 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:58:28.945864 7066 webserver.cc:460] Webserver started at http://127.21.159.194:41897/ using document root <none> and password file <none>
I20260620 14:58:28.946585 7066 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:28.946719 7066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:28.947119 7066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:28.950201 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "6438942d585a44d6a19f53211905c5aa"
format_stamp: "Formatted at 2026-06-20 14:58:28 on dist-test-slave-3mqx"
server_key: "ba9af07cbc862090463fdaa896994291"
server_key_iv: "11d11d9aa44c53a624d6325a714b1945"
server_key_version: "encryptionkey@0"
I20260620 14:58:28.951031 7066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "6438942d585a44d6a19f53211905c5aa"
format_stamp: "Formatted at 2026-06-20 14:58:28 on dist-test-slave-3mqx"
server_key: "ba9af07cbc862090463fdaa896994291"
server_key_iv: "11d11d9aa44c53a624d6325a714b1945"
server_key_version: "encryptionkey@0"
I20260620 14:58:28.956627 7066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:58:28.960798 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:28.962392 7066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:28.962589 7066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "6438942d585a44d6a19f53211905c5aa"
format_stamp: "Formatted at 2026-06-20 14:58:28 on dist-test-slave-3mqx"
server_key: "ba9af07cbc862090463fdaa896994291"
server_key_iv: "11d11d9aa44c53a624d6325a714b1945"
server_key_version: "encryptionkey@0"
I20260620 14:58:28.962814 7066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:29.000388 7066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:29.005541 7066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:29.005898 7066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:29.006966 7066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:29.008497 7066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:29.008581 7066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:29.008906 7066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:29.008985 7066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:29.045835 7066 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:39241
I20260620 14:58:29.045923 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:39241 every 8 connection(s)
I20260620 14:58:29.047309 7066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:29.051787 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 7066
I20260620 14:58:29.052037 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:29.052659 22143 external_mini_cluster.cc:1428] Setting key 90b0da5696ac0aba6c15f082bcb368bb
Jun 20 14:58:29 dist-test-slave-3mqx krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967508, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:29.071676 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.050246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:58001 (local address 127.21.159.254:32769)
0620 14:58:29.050702 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.050706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.052542 (+ 1836us) server_negotiation.cc:407] Connection header received
0620 14:58:29.054027 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.054043 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.054119 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.054235 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:29.055977 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.057255 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.058787 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.059582 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.059727 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.063687 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:29.063693 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:29.063696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:29.063732 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:29.067710 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:29.068862 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:29.068867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:29.068869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:29.069037 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:29.069654 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:29.069661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:29.069664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:29.069967 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:29.070093 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:29.071189 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0620 14:58:29.071402 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260620 14:58:29.073297 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.050950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32769 (local address 127.21.159.194:58001)
0620 14:58:29.051792 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.051845 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.053679 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.054528 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.054540 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.055196 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:29.055775 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.055797 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.057408 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.057412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.058637 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.058649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.059788 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.059793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.060090 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.060839 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:29.060863 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:29.063438 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:29.067916 (+ 4478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:29.067929 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:29.067951 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:29.068657 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:29.069170 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:29.069177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:29.069184 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:29.069467 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:29.070113 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:29.070125 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:29.070630 (+ 505us) client_negotiation.cc:769] Sending connection context
0620 14:58:29.070983 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 14:58:29.071671 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":351,"threads_started":1}
I20260620 14:58:29.074911 7194 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32769
I20260620 14:58:29.075289 7194 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:29.076311 7194 heartbeater.cc:511] Master 127.21.159.254:32769 requested a full tablet report, sending...
I20260620 14:58:29.078401 6871 ts_manager.cc:194] Registered new tserver with Master: 6438942d585a44d6a19f53211905c5aa (127.21.159.194:39241)
I20260620 14:58:29.079480 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:58001
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:58:29.115433 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32769
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:29.461516 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:29.461861 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:29.462034 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:29.469908 7200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:29.470077 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:29.470257 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:58:29.480629 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:29.482028 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:29.483453 7200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:29.497313 7205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:29.497244 7208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:29.497282 7206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:29.498200 7200 server_base.cc:1034] running on GCE node
I20260620 14:58:29.498831 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:29.499912 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:29.501155 7200 hybrid_clock.cc:648] HybridClock initialized: now 1781967509501066 us; error 60 us; skew 500 ppm
I20260620 14:58:29.501536 7200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:29 dist-test-slave-3mqx krb5kdc[6826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967509, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:29.505101 7200 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:58:29.506747 7200 webserver.cc:460] Webserver started at http://127.21.159.195:33171/ using document root <none> and password file <none>
I20260620 14:58:29.507527 7200 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:29.507678 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:29.507998 7200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:29.510970 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "6a6254ba56024b81825065fab9590d14"
format_stamp: "Formatted at 2026-06-20 14:58:29 on dist-test-slave-3mqx"
server_key: "9f593127e3e67eb37b604b4b31e83bbb"
server_key_iv: "f086c1011f9bf4c348eee4519e9f199d"
server_key_version: "encryptionkey@0"
I20260620 14:58:29.511802 7200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "6a6254ba56024b81825065fab9590d14"
format_stamp: "Formatted at 2026-06-20 14:58:29 on dist-test-slave-3mqx"
server_key: "9f593127e3e67eb37b604b4b31e83bbb"
server_key_iv: "f086c1011f9bf4c348eee4519e9f199d"
server_key_version: "encryptionkey@0"
I20260620 14:58:29.517364 7200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 14:58:29.521292 7215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:29.522778 7200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:29.522969 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "6a6254ba56024b81825065fab9590d14"
format_stamp: "Formatted at 2026-06-20 14:58:29 on dist-test-slave-3mqx"
server_key: "9f593127e3e67eb37b604b4b31e83bbb"
server_key_iv: "f086c1011f9bf4c348eee4519e9f199d"
server_key_version: "encryptionkey@0"
I20260620 14:58:29.523273 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:29.529281 7060 heartbeater.cc:503] Master 127.21.159.254:32769 was elected leader, sending a full tablet report...
I20260620 14:58:29.549259 7200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:29.553593 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:29.553854 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:29.554828 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:29.556227 7200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:29.556315 7200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:29.556402 7200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:29.556447 7200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:29.595553 7200 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:45573
I20260620 14:58:29.595664 7327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:45573 every 8 connection(s)
I20260620 14:58:29.597008 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:29.602182 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 7200
I20260620 14:58:29.602407 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:29.603003 22143 external_mini_cluster.cc:1428] Setting key b5731b0dc9cc5499514a61611bc21191
Jun 20 14:58:29 dist-test-slave-3mqx krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967509, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:29.620507 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.600084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:41035 (local address 127.21.159.254:32769)
0620 14:58:29.600574 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.600579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.601695 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 14:58:29.604383 (+ 2688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.604389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.604489 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.604681 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:29.606703 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.607674 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.609286 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.609714 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.609816 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.613476 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:29.613481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:29.613484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:29.613519 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:29.616418 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:29.617433 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:29.617438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:29.617441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:29.617605 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:29.618183 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:29.618189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:29.618192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:29.618535 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:29.618725 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:29.619773 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0620 14:58:29.620039 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":236,"threads_started":1}
I20260620 14:58:29.621690 7331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.600712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32769 (local address 127.21.159.195:41035)
0620 14:58:29.601506 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.601549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.604036 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.605028 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.605043 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.605794 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:29.606493 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.606528 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.607838 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.607842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.609138 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.609152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.609901 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.609905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.610097 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.610817 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:29.610841 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:29.613249 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:29.616603 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:29.616611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:29.616628 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:29.617294 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:29.617748 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:29.617754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:29.617758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:29.618020 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:29.618718 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:29.618750 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:29.619298 (+ 548us) client_negotiation.cc:769] Sending connection context
0620 14:58:29.619685 (+ 387us) client_negotiation.cc:240] Negotiation successful
0620 14:58:29.620371 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":303,"threads_started":1}
I20260620 14:58:29.623131 7328 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32769
I20260620 14:58:29.623486 7328 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:29.624233 7328 heartbeater.cc:511] Master 127.21.159.254:32769 requested a full tablet report, sending...
I20260620 14:58:29.625694 6871 ts_manager.cc:194] Registered new tserver with Master: 6a6254ba56024b81825065fab9590d14 (127.21.159.195:45573)
I20260620 14:58:29.626554 6871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:41035
I20260620 14:58:29.632035 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:29.651434 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.636907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.21.159.254:32769)
0620 14:58:29.637096 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.637104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.637640 (+ 536us) server_negotiation.cc:407] Connection header received
0620 14:58:29.637803 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.637808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.637874 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.637961 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:29.639658 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.640459 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.642073 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.642490 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.642606 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.645374 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:29.645379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:29.645383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:29.645421 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:29.648461 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:29.649345 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:29.649350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:29.649352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:29.649509 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:29.649909 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:29.649913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:29.649915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:29.650334 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:29.650545 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:29.650901 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:58:29.651186 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:58:29.656023 6871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:29.663388 6871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:29.689272 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.676642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:39241 (local address 127.0.0.1:56272)
0620 14:58:29.677583 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.677641 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.677883 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.680264 (+ 2381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.680271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.680310 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:29.680911 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.680925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.683568 (+ 2643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.683578 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.686493 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.686506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.688134 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.688140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.688357 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.688413 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 14:58:29.688556 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 14:58:29.688749 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":55,"thread_start_us":1575,"threads_started":1}
I20260620 14:58:29.689693 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.676797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.21.159.194:39241)
0620 14:58:29.679576 (+ 2779us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.679589 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.679620 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:58:29.679727 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.679735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.679979 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.681145 (+ 1166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:29.681382 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.683374 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.686647 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.687928 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.688123 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.688305 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:29.689158 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 14:58:29.689356 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":2764,"threads_started":1}
I20260620 14:58:29.690732 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.677239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.21.159.195:45573)
0620 14:58:29.679241 (+ 2002us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.679250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.679287 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:58:29.679387 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.679397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.679665 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.680835 (+ 1170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:29.681183 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.683334 (+ 2151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.687034 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.688969 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.690077 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.690230 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:29.690285 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:58:29.690403 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":1642,"threads_started":1}
I20260620 14:58:29.691196 7341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.676798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:45573 (local address 127.0.0.1:46922)
0620 14:58:29.678547 (+ 1749us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.678569 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.678662 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.679975 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.679987 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.680029 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:29.680558 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.680575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.683536 (+ 2961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.683546 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.686867 (+ 3321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.686883 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.689285 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.689294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.689499 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.689533 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:58:29.689614 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:58:29.689751 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1592,"mutex_wait_us":791,"thread_start_us":767,"threads_started":1}
I20260620 14:58:29.694295 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.679447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.0.0.1:56716)
0620 14:58:29.684808 (+ 5361us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:29.684825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:29.684937 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:29.686692 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:29.686696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:29.686730 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:29.687155 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.687171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.689119 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.689124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.691319 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:29.691338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.693500 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.693509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:29.693692 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.693735 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 14:58:29.693835 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:58:29.693968 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5235,"thread_start_us":5079,"threads_started":1}
I20260620 14:58:29.694438 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:29.679823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.21.159.193:43989)
0620 14:58:29.686092 (+ 6269us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:29.686101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:29.686128 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:58:29.686217 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:29.686225 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:29.686502 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:29.686683 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:29.687418 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.688942 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.691501 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:29.692985 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:29.693186 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:29.693356 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:29.693931 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 14:58:29.694095 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6098,"thread_start_us":5919,"threads_started":1}
I20260620 14:58:29.694208 7129 tablet_service.cc:1467] Processing CreateTablet for tablet f19dd8d09238446ab08f8440100c86d6 (DEFAULT_TABLE table=test-table [id=d52502d75eb04b67a2625fd04ed2d94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:29.695739 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19dd8d09238446ab08f8440100c86d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:29.695468 7263 tablet_service.cc:1467] Processing CreateTablet for tablet f19dd8d09238446ab08f8440100c86d6 (DEFAULT_TABLE table=test-table [id=d52502d75eb04b67a2625fd04ed2d94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:29.696768 7263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19dd8d09238446ab08f8440100c86d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:29.698190 6995 tablet_service.cc:1467] Processing CreateTablet for tablet f19dd8d09238446ab08f8440100c86d6 (DEFAULT_TABLE table=test-table [id=d52502d75eb04b67a2625fd04ed2d94f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:29.699617 6995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f19dd8d09238446ab08f8440100c86d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:29.708122 7347 tablet_bootstrap.cc:492] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Bootstrap starting.
I20260620 14:58:29.710695 7348 tablet_bootstrap.cc:492] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Bootstrap starting.
I20260620 14:58:29.712972 7347 tablet_bootstrap.cc:654] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:29.714217 7349 tablet_bootstrap.cc:492] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Bootstrap starting.
I20260620 14:58:29.714344 7347 log.cc:824] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:29.715669 7348 tablet_bootstrap.cc:654] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:29.717020 7348 log.cc:824] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:29.718081 7349 tablet_bootstrap.cc:654] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:29.718223 7347 tablet_bootstrap.cc:492] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: No bootstrap required, opened a new log
I20260620 14:58:29.718542 7347 ts_tablet_manager.cc:1397] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 14:58:29.719264 7349 log.cc:824] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:29.720341 7348 tablet_bootstrap.cc:492] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: No bootstrap required, opened a new log
I20260620 14:58:29.720631 7348 ts_tablet_manager.cc:1397] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260620 14:58:29.722127 7349 tablet_bootstrap.cc:492] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: No bootstrap required, opened a new log
I20260620 14:58:29.722397 7349 ts_tablet_manager.cc:1397] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 14:58:29.730043 7349 raft_consensus.cc:348] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:29.730538 7349 raft_consensus.cc:374] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:29.730643 7349 raft_consensus.cc:729] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a781dcf039d148db81152aa9534c4b02, State: Initialized, Role: FOLLOWER
I20260620 14:58:29.730775 7347 raft_consensus.cc:348] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:29.731456 7347 raft_consensus.cc:374] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:29.731566 7347 raft_consensus.cc:729] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6254ba56024b81825065fab9590d14, State: Initialized, Role: FOLLOWER
I20260620 14:58:29.731393 7349 consensus_queue.cc:260] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:29.731519 7348 raft_consensus.cc:348] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:29.731961 7348 raft_consensus.cc:374] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:29.732070 7348 raft_consensus.cc:729] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6438942d585a44d6a19f53211905c5aa, State: Initialized, Role: FOLLOWER
I20260620 14:58:29.732676 7348 consensus_queue.cc:260] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:29.732676 7347 consensus_queue.cc:260] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:29.734025 7328 heartbeater.cc:503] Master 127.21.159.254:32769 was elected leader, sending a full tablet report...
I20260620 14:58:29.735343 7349 ts_tablet_manager.cc:1428] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260620 14:58:29.736503 7194 heartbeater.cc:503] Master 127.21.159.254:32769 was elected leader, sending a full tablet report...
I20260620 14:58:29.736646 7347 ts_tablet_manager.cc:1428] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260620 14:58:29.738045 7348 ts_tablet_manager.cc:1428] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
W20260620 14:58:29.742795 7061 tablet.cc:2369] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:29.799603 7195 tablet.cc:2369] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:29.848871 7329 tablet.cc:2369] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:30.055668 7354 raft_consensus.cc:482] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:30.055980 7354 raft_consensus.cc:504] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:30.057938 7354 leader_election.cc:288] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a6254ba56024b81825065fab9590d14 (127.21.159.195:45573), a781dcf039d148db81152aa9534c4b02 (127.21.159.193:43989)
I20260620 14:58:30.065851 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.058429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:39495 (local address 127.21.159.195:45573)
0620 14:58:30.058640 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.058649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.059554 (+ 905us) server_negotiation.cc:407] Connection header received
0620 14:58:30.059727 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.059731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.059789 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.059916 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:30.060597 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.061700 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.063800 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.064718 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.064880 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.064960 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:30.065463 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 14:58:30.065598 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:58:30.065841 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.058393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:52409 (local address 127.21.159.193:43989)
0620 14:58:30.058664 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.058673 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.059470 (+ 797us) server_negotiation.cc:407] Connection header received
0620 14:58:30.059660 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.059669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.059757 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.059824 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:30.060470 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.061677 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.063833 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.064606 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.064773 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.064873 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:30.065395 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 14:58:30.065552 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:58:30.066013 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.058209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:43989 (local address 127.21.159.194:52409)
0620 14:58:30.059347 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:30.059363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:30.059485 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:30.059883 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:30.059891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:30.059927 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:30.060316 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.060325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.061835 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.061839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.063630 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.063642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.064844 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.064850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.065078 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.065202 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 14:58:30.065333 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:58:30.065530 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":180,"thread_start_us":808,"threads_started":1}
I20260620 14:58:30.066087 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.058418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:45573 (local address 127.21.159.194:39495)
0620 14:58:30.059399 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:30.059439 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:30.059599 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:30.059947 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:30.059952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:30.059984 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:30.060444 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.060453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.061856 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.061859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.063637 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:30.063649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.064969 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.064974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:30.065170 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.065226 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 14:58:30.065411 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:58:30.065579 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":118,"thread_start_us":459,"threads_started":1}
I20260620 14:58:30.067158 7015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19dd8d09238446ab08f8440100c86d6" candidate_uuid: "6438942d585a44d6a19f53211905c5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a781dcf039d148db81152aa9534c4b02" is_pre_election: true
I20260620 14:58:30.067723 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19dd8d09238446ab08f8440100c86d6" candidate_uuid: "6438942d585a44d6a19f53211905c5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6254ba56024b81825065fab9590d14" is_pre_election: true
I20260620 14:58:30.067902 7015 raft_consensus.cc:2454] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6438942d585a44d6a19f53211905c5aa in term 0.
I20260620 14:58:30.068351 7283 raft_consensus.cc:2454] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6438942d585a44d6a19f53211905c5aa in term 0.
I20260620 14:58:30.068684 7084 leader_election.cc:302] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6438942d585a44d6a19f53211905c5aa, a781dcf039d148db81152aa9534c4b02; no voters:
I20260620 14:58:30.069118 7354 raft_consensus.cc:2781] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:30.069284 7354 raft_consensus.cc:482] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:30.069427 7354 raft_consensus.cc:3037] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:30.071972 7354 raft_consensus.cc:504] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:30.072726 7354 leader_election.cc:288] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [CANDIDATE]: Term 1 election: Requested vote from peers 6a6254ba56024b81825065fab9590d14 (127.21.159.195:45573), a781dcf039d148db81152aa9534c4b02 (127.21.159.193:43989)
I20260620 14:58:30.073217 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19dd8d09238446ab08f8440100c86d6" candidate_uuid: "6438942d585a44d6a19f53211905c5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6254ba56024b81825065fab9590d14"
I20260620 14:58:30.073298 7015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f19dd8d09238446ab08f8440100c86d6" candidate_uuid: "6438942d585a44d6a19f53211905c5aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a781dcf039d148db81152aa9534c4b02"
I20260620 14:58:30.073485 7283 raft_consensus.cc:3037] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:30.073526 7015 raft_consensus.cc:3037] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:30.075922 7015 raft_consensus.cc:2454] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6438942d585a44d6a19f53211905c5aa in term 1.
I20260620 14:58:30.076102 7283 raft_consensus.cc:2454] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6438942d585a44d6a19f53211905c5aa in term 1.
I20260620 14:58:30.076400 7084 leader_election.cc:302] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6438942d585a44d6a19f53211905c5aa, a781dcf039d148db81152aa9534c4b02; no voters:
I20260620 14:58:30.076807 7354 raft_consensus.cc:2781] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:30.077394 7354 raft_consensus.cc:686] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 1 LEADER]: Becoming Leader. State: Replica: 6438942d585a44d6a19f53211905c5aa, State: Running, Role: LEADER
I20260620 14:58:30.077960 7354 consensus_queue.cc:237] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } }
I20260620 14:58:30.083395 6870 catalog_manager.cc:5495] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 6438942d585a44d6a19f53211905c5aa (127.21.159.194). New cstate: current_term: 1 leader_uuid: "6438942d585a44d6a19f53211905c5aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6438942d585a44d6a19f53211905c5aa" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 } health_report { overall_health: UNKNOWN } } }
I20260620 14:58:30.150529 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.144945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.21.159.194:39241)
0620 14:58:30.145125 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.145129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.145147 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:58:30.145410 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.145413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.145494 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.145631 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:30.146084 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.146845 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.148498 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.148908 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.149005 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.149474 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:30.150011 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:30.150122 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:30.150236 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:58:30.150306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:58:30.159693 7283 raft_consensus.cc:1261] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 1 FOLLOWER]: Refusing update from remote peer 6438942d585a44d6a19f53211905c5aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:30.159695 7015 raft_consensus.cc:1261] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 1 FOLLOWER]: Refusing update from remote peer 6438942d585a44d6a19f53211905c5aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:30.160753 7358 consensus_queue.cc:1035] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6254ba56024b81825065fab9590d14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 45573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:30.161587 7354 consensus_queue.cc:1035] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a781dcf039d148db81152aa9534c4b02" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 43989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:30.185102 7363 mvcc.cc:205] Tried to move back new op lower bound from 7298938921602838528 to 7298938921283371008. Current Snapshot: MvccSnapshot[applied={T|T < 7298938921602838528}]
I20260620 14:58:30.192695 6871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:30.192973 6871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:30.196723 6871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51c7c2a6b13742378eb24a63cfda26d2: Sending DeleteTablet for 3 replicas of tablet f19dd8d09238446ab08f8440100c86d6
I20260620 14:58:30.198210 7263 tablet_service.cc:1514] Processing DeleteTablet for tablet f19dd8d09238446ab08f8440100c86d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:30 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:46922
I20260620 14:58:30.199277 6995 tablet_service.cc:1514] Processing DeleteTablet for tablet f19dd8d09238446ab08f8440100c86d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:30 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56716
I20260620 14:58:30.199662 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 6932
I20260620 14:58:30.201179 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet f19dd8d09238446ab08f8440100c86d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:30 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:56272
I20260620 14:58:30.207932 7373 tablet_replica.cc:330] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: stopping tablet replica
I20260620 14:58:30.208588 7373 raft_consensus.cc:2229] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:30.209566 7373 raft_consensus.cc:2258] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:30.214672 7371 tablet_replica.cc:330] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: stopping tablet replica
I20260620 14:58:30.215201 7371 raft_consensus.cc:2229] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:30.215758 7371 raft_consensus.cc:2258] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:30.217120 7373 ts_tablet_manager.cc:1905] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:30.217993 7371 ts_tablet_manager.cc:1905] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:30.225172 7371 ts_tablet_manager.cc:1918] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:30.225373 7371 log.cc:1196] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/f19dd8d09238446ab08f8440100c86d6
I20260620 14:58:30.226424 7372 tablet_replica.cc:330] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: stopping tablet replica
I20260620 14:58:30.226938 7372 raft_consensus.cc:2229] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:30.227562 7372 raft_consensus.cc:2258] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:30.225921 7371 ts_tablet_manager.cc:1939] T f19dd8d09238446ab08f8440100c86d6 P 6a6254ba56024b81825065fab9590d14: Deleting consensus metadata
I20260620 14:58:30.229506 7372 ts_tablet_manager.cc:1905] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 14:58:30 dist-test-slave-3mqx krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:58:30.230895 7373 ts_tablet_manager.cc:1918] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:30.231134 7373 log.cc:1196] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/f19dd8d09238446ab08f8440100c86d6
I20260620 14:58:30.231810 7373 ts_tablet_manager.cc:1939] T f19dd8d09238446ab08f8440100c86d6 P 6438942d585a44d6a19f53211905c5aa: Deleting consensus metadata
I20260620 14:58:30.234869 6857 catalog_manager.cc:4841] TS 6438942d585a44d6a19f53211905c5aa (127.21.159.194:39241): tablet f19dd8d09238446ab08f8440100c86d6 (table test-table [id=d52502d75eb04b67a2625fd04ed2d94f]) successfully deleted
I20260620 14:58:30.236977 7372 ts_tablet_manager.cc:1918] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:30.237156 7372 log.cc:1196] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/f19dd8d09238446ab08f8440100c86d6
I20260620 14:58:30.237838 7372 ts_tablet_manager.cc:1939] T f19dd8d09238446ab08f8440100c86d6 P a781dcf039d148db81152aa9534c4b02: Deleting consensus metadata
I20260620 14:58:30.237869 6855 catalog_manager.cc:4841] TS 6a6254ba56024b81825065fab9590d14 (127.21.159.195:45573): tablet f19dd8d09238446ab08f8440100c86d6 (table test-table [id=d52502d75eb04b67a2625fd04ed2d94f]) successfully deleted
I20260620 14:58:30.239568 6857 catalog_manager.cc:4841] TS a781dcf039d148db81152aa9534c4b02 (127.21.159.193:43989): tablet f19dd8d09238446ab08f8440100c86d6 (table test-table [id=d52502d75eb04b67a2625fd04ed2d94f]) successfully deleted
I20260620 14:58:30.240512 7346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.202743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.21.159.193:43989)
0620 14:58:30.202992 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.203001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.214876 (+ 11875us) server_negotiation.cc:407] Connection header received
0620 14:58:30.215783 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.215794 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.215948 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.216059 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:30.218521 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.219871 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.222571 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.223633 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.223767 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.231476 (+ 7709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:30.231496 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:30.231517 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:30.231588 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:30.236408 (+ 4820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:30.237687 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:30.237701 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:30.237705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:30.237983 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:30.238604 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:30.238613 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:30.238618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:30.239031 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:30.239433 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:30.239887 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 14:58:30.240134 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:58:30.241186 7055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:30.337980 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 6932
I20260620 14:58:30.355998 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 7066
Jun 20 14:58:30 dist-test-slave-3mqx krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:58:30.374951 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.356962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.21.159.194:39241)
0620 14:58:30.357192 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.357199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.357222 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:58:30.357287 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.357293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.357401 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.357519 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:30.358895 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.360164 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.361652 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.362371 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.362513 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.367231 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:30.367255 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:30.367277 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:30.367348 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:30.371529 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:30.372530 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:30.372538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:30.372541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:30.372724 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:30.373189 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:30.373195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:30.373198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:30.373654 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:30.373963 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:30.374373 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 14:58:30.374559 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:58:30.375761 7189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:30.498451 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 7066
I20260620 14:58:30.517834 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 7200
Jun 20 14:58:30 dist-test-slave-3mqx krb5kdc[6826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:58:30.534175 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:30.518667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.21.159.195:45573)
0620 14:58:30.518929 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:30.518937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:30.518963 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:58:30.519253 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:30.519264 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:30.519353 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:30.519463 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:30.520819 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.521538 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.522817 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:30.523283 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:30.523418 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:30.528216 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:30.528234 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:30.528251 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:30.528289 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:30.531306 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:30.532071 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:30.532079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:30.532082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:30.532268 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:30.532701 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:30.532705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:30.532708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:30.533102 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:30.533355 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:30.533712 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:58:30.533888 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 14:58:30.534936 7323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:30.628181 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 7200
I20260620 14:58:30.644940 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 6839
I20260620 14:58:30.646152 6901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:30.758085 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 6839
2026-06-20T14:58:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4821 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:30Z Disabled control of system clock
I20260620 14:58:30.840029 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:39767
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:39767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:58:31.181648 7382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:31.181957 7382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:31.182040 7382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:31.189599 7382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:31.189728 7382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:31.189766 7382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:31.189817 7382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:31.189862 7382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:31.199651 7382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:39767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:39767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:31.201012 7382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:31.202822 7382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:31.215430 7387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:31.215353 7388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:31.215401 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:31.216316 7382 server_base.cc:1034] running on GCE node
I20260620 14:58:31.217150 7382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:31.218677 7382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:31.219893 7382 hybrid_clock.cc:648] HybridClock initialized: now 1781967511219850 us; error 39 us; skew 500 ppm
I20260620 14:58:31.220292 7382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:31.222652 7382 webserver.cc:460] Webserver started at http://127.21.159.254:42917/ using document root <none> and password file <none>
I20260620 14:58:31.223433 7382 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:31.223523 7382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:31.223873 7382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:31.227221 7382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "7a5c08521418468bb144aa0cd06be5cc"
format_stamp: "Formatted at 2026-06-20 14:58:31 on dist-test-slave-3mqx"
server_key: "c1609930d2a3cecebdb7c760f06e9a7e"
server_key_iv: "02e9013322cb6b2b2a297997c3bd5e34"
server_key_version: "encryptionkey@0"
I20260620 14:58:31.228082 7382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "7a5c08521418468bb144aa0cd06be5cc"
format_stamp: "Formatted at 2026-06-20 14:58:31 on dist-test-slave-3mqx"
server_key: "c1609930d2a3cecebdb7c760f06e9a7e"
server_key_iv: "02e9013322cb6b2b2a297997c3bd5e34"
server_key_version: "encryptionkey@0"
I20260620 14:58:31.234017 7382 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:58:31.237999 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:31.239787 7382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:31.240049 7382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "7a5c08521418468bb144aa0cd06be5cc"
format_stamp: "Formatted at 2026-06-20 14:58:31 on dist-test-slave-3mqx"
server_key: "c1609930d2a3cecebdb7c760f06e9a7e"
server_key_iv: "02e9013322cb6b2b2a297997c3bd5e34"
server_key_version: "encryptionkey@0"
I20260620 14:58:31.240336 7382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:31.272743 7382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:31.273823 7382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:31.274169 7382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:31.295980 7447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:39767 every 8 connection(s)
I20260620 14:58:31.295975 7382 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:39767
I20260620 14:58:31.297600 7382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:31.301761 7448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:31.305164 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 7382
I20260620 14:58:31.305395 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:31.306002 22143 external_mini_cluster.cc:1428] Setting key eb4ab31af889e4e4979ded4ada44b054
I20260620 14:58:31.315680 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc: Bootstrap starting.
I20260620 14:58:31.320056 7448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:31.321383 7448 log.cc:824] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:31.322171 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:31.308015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.21.159.254:39767)
0620 14:58:31.309059 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:31.309071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:31.309117 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:58:31.311014 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:31.311041 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:31.311659 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:31.312034 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:31.312818 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.314222 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.316455 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.317270 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.317429 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.318999 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:31.319007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:31.319022 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:31.319759 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:31.319902 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:31.319937 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:31.320080 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:31.320477 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 14:58:31.320897 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":359,"threads_started":1}
I20260620 14:58:31.325223 7448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc: No bootstrap required, opened a new log
I20260620 14:58:31.333528 7448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5c08521418468bb144aa0cd06be5cc" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39767 } }
I20260620 14:58:31.333926 7448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:31.334005 7448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5c08521418468bb144aa0cd06be5cc, State: Initialized, Role: FOLLOWER
I20260620 14:58:31.334539 7448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5c08521418468bb144aa0cd06be5cc" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39767 } }
I20260620 14:58:31.334764 7448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:31.334857 7448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:31.334975 7448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:31.337594 7448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5c08521418468bb144aa0cd06be5cc" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39767 } }
I20260620 14:58:31.338253 7448 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5c08521418468bb144aa0cd06be5cc; no voters:
I20260620 14:58:31.338891 7448 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:31.339108 7453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:31.339949 7453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [term 1 LEADER]: Becoming Leader. State: Replica: 7a5c08521418468bb144aa0cd06be5cc, State: Running, Role: LEADER
I20260620 14:58:31.340606 7453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5c08521418468bb144aa0cd06be5cc" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39767 } }
I20260620 14:58:31.341243 7448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:31.343807 7454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5c08521418468bb144aa0cd06be5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5c08521418468bb144aa0cd06be5cc" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39767 } } }
I20260620 14:58:31.344203 7454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:31.345185 7460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:31.347942 7455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5c08521418468bb144aa0cd06be5cc. Latest consensus state: current_term: 1 leader_uuid: "7a5c08521418468bb144aa0cd06be5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5c08521418468bb144aa0cd06be5cc" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 39767 } } }
I20260620 14:58:31.348820 7455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:31.352360 7460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:31.371395 7460 catalog_manager.cc:1347] Generated new cluster ID: a341ca68d21149fa8605174d2dbe42c3
I20260620 14:58:31.371567 7460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:31.390753 7460 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:31.393061 7460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:31.403030 7460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc: Generated new TSK 0
I20260620 14:58:31.404264 7460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:31.409579 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39767
--builtin_ntp_servers=127.21.159.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:58:31.760916 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:31.761219 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:31.761307 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:31.768849 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:31.768965 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:31.769071 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:58:31.778914 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:31.780263 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:31.782737 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:31.797204 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:31.797261 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:31.797183 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:31.798069 7472 server_base.cc:1034] running on GCE node
I20260620 14:58:31.798811 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:31.799999 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:31.801256 7472 hybrid_clock.cc:648] HybridClock initialized: now 1781967511801167 us; error 46 us; skew 500 ppm
I20260620 14:58:31.801618 7472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:31.804343 7472 webserver.cc:460] Webserver started at http://127.21.159.193:37467/ using document root <none> and password file <none>
I20260620 14:58:31.805181 7472 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:31.805286 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:31.805665 7472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:31.808876 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "33316cc0e99d4080b368feecdee35679"
format_stamp: "Formatted at 2026-06-20 14:58:31 on dist-test-slave-3mqx"
server_key: "05cab9f6e2ec65b2571163d8e1e632d1"
server_key_iv: "efe691030136aff3aade8e5faba1d4f7"
server_key_version: "encryptionkey@0"
I20260620 14:58:31.809782 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "33316cc0e99d4080b368feecdee35679"
format_stamp: "Formatted at 2026-06-20 14:58:31 on dist-test-slave-3mqx"
server_key: "05cab9f6e2ec65b2571163d8e1e632d1"
server_key_iv: "efe691030136aff3aade8e5faba1d4f7"
server_key_version: "encryptionkey@0"
I20260620 14:58:31.815394 7472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:58:31.819352 7486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:31.821045 7472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:31.821339 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "33316cc0e99d4080b368feecdee35679"
format_stamp: "Formatted at 2026-06-20 14:58:31 on dist-test-slave-3mqx"
server_key: "05cab9f6e2ec65b2571163d8e1e632d1"
server_key_iv: "efe691030136aff3aade8e5faba1d4f7"
server_key_version: "encryptionkey@0"
I20260620 14:58:31.821599 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:31.839167 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:31.840092 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:31.840425 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:31.841471 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:31.843365 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:31.843487 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:31.843622 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:31.843720 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:31.877430 7472 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:46379
I20260620 14:58:31.877476 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:46379 every 8 connection(s)
I20260620 14:58:31.878901 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:31.885787 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 7472
I20260620 14:58:31.886006 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:31.886516 22143 external_mini_cluster.cc:1428] Setting key 2fe093dcc8c64f987d3b49f2cbcc18fb
I20260620 14:58:31.889441 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39767
--builtin_ntp_servers=127.21.159.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:58:31.892629 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:31.881716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:38693 (local address 127.21.159.254:39767)
0620 14:58:31.882200 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:31.882206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:31.883590 (+ 1384us) server_negotiation.cc:407] Connection header received
0620 14:58:31.885447 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:31.885454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:31.885557 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:31.885710 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:31.886975 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.887887 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.889752 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.890145 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.890292 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.891747 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:31.891755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:31.891781 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:31.891799 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:31.891839 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:31.891851 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:31.891953 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:31.892175 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:58:31.892281 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":217,"threads_started":1}
I20260620 14:58:31.893676 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:31.882427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39767 (local address 127.21.159.193:38693)
0620 14:58:31.883420 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:31.883465 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:31.885110 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:31.885952 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:31.885971 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:31.886099 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:31.886796 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.886815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.888046 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.888051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.889560 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:31.889579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:31.890396 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:31.890403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:31.890638 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:31.891416 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:31.891419 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:31.891459 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:31.891541 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:31.891952 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:31.891965 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:31.891976 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:58:31.892167 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 14:58:31.892585 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":291,"threads_started":1}
I20260620 14:58:31.895452 7599 heartbeater.cc:347] Connected to a master server at 127.21.159.254:39767
I20260620 14:58:31.895790 7599 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:31.896657 7599 heartbeater.cc:511] Master 127.21.159.254:39767 requested a full tablet report, sending...
I20260620 14:58:31.898850 7413 ts_manager.cc:194] Registered new tserver with Master: 33316cc0e99d4080b368feecdee35679 (127.21.159.193:46379)
I20260620 14:58:31.900435 7413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.193:38693
W20260620 14:58:32.232066 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:32.232448 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:32.232627 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:32.240491 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:32.240651 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:32.240841 7603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:58:32.250823 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:32.252278 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:32.254326 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:32.269212 7611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:32.269181 7609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:32.269156 7608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:32.269956 7603 server_base.cc:1034] running on GCE node
I20260620 14:58:32.270535 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:32.271567 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:32.272830 7603 hybrid_clock.cc:648] HybridClock initialized: now 1781967512272769 us; error 53 us; skew 500 ppm
I20260620 14:58:32.273205 7603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:32.275866 7603 webserver.cc:460] Webserver started at http://127.21.159.194:33501/ using document root <none> and password file <none>
I20260620 14:58:32.276604 7603 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:32.276743 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:32.277072 7603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:32.280167 7603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "f738895d02214465962e1ee5b82cc0f2"
format_stamp: "Formatted at 2026-06-20 14:58:32 on dist-test-slave-3mqx"
server_key: "8ac2e1590c5706acf2587924529553ce"
server_key_iv: "681d4ba3725ca960b058ae17873cb2d6"
server_key_version: "encryptionkey@0"
I20260620 14:58:32.281005 7603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "f738895d02214465962e1ee5b82cc0f2"
format_stamp: "Formatted at 2026-06-20 14:58:32 on dist-test-slave-3mqx"
server_key: "8ac2e1590c5706acf2587924529553ce"
server_key_iv: "681d4ba3725ca960b058ae17873cb2d6"
server_key_version: "encryptionkey@0"
I20260620 14:58:32.286388 7603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:32.290122 7617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:32.291733 7603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:32.291921 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "f738895d02214465962e1ee5b82cc0f2"
format_stamp: "Formatted at 2026-06-20 14:58:32 on dist-test-slave-3mqx"
server_key: "8ac2e1590c5706acf2587924529553ce"
server_key_iv: "681d4ba3725ca960b058ae17873cb2d6"
server_key_version: "encryptionkey@0"
I20260620 14:58:32.292155 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:32.336699 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:32.337597 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:32.337935 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:32.339028 7603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:32.340657 7603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:32.340786 7603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:32.340940 7603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:32.341044 7603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 14:58:32.377955 7603 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:34813
I20260620 14:58:32.378010 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:34813 every 8 connection(s)
I20260620 14:58:32.379634 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:32.386533 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 7603
I20260620 14:58:32.386771 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:32.387331 22143 external_mini_cluster.cc:1428] Setting key a0e8cb73267d2c86d872530e78bf79e4
I20260620 14:58:32.390750 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39767
--builtin_ntp_servers=127.21.159.212:37287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:58:32.397827 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.382933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:55073 (local address 127.21.159.254:39767)
0620 14:58:32.383276 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.383281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.384876 (+ 1595us) server_negotiation.cc:407] Connection header received
0620 14:58:32.387159 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.387167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.387291 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.387441 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:32.389023 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.390082 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.392898 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.393947 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.394179 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.396515 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:32.396546 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:32.396556 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:32.396588 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:32.396660 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:32.396683 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:32.396816 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:32.397228 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 14:58:32.397357 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":85}
I20260620 14:58:32.399673 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.383554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39767 (local address 127.21.159.194:55073)
0620 14:58:32.384685 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.384735 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.386658 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.387728 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.387738 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.387884 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:32.388805 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.388834 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.390338 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.390343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.392676 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.392701 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.394288 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.394300 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.394731 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.395871 (+ 1140us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:32.395878 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:32.395951 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:32.396231 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:32.396858 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:32.396874 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:32.396886 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:58:32.397140 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 14:58:32.397813 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":326,"threads_started":1}
I20260620 14:58:32.401641 7730 heartbeater.cc:347] Connected to a master server at 127.21.159.254:39767
I20260620 14:58:32.402001 7730 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:32.402854 7730 heartbeater.cc:511] Master 127.21.159.254:39767 requested a full tablet report, sending...
I20260620 14:58:32.404556 7413 ts_manager.cc:194] Registered new tserver with Master: f738895d02214465962e1ee5b82cc0f2 (127.21.159.194:34813)
I20260620 14:58:32.405721 7413 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.194:55073
W20260620 14:58:32.739271 7733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:32.739638 7733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:32.739817 7733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:32.748272 7733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:32.748431 7733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:32.748618 7733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:58:32.758550 7733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:37287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:39767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:32.760052 7733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:32.761883 7733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:32.776649 7739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:32.776516 7738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:32.776978 7741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:32.777853 7733 server_base.cc:1034] running on GCE node
I20260620 14:58:32.778544 7733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:32.779835 7733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:32.781189 7733 hybrid_clock.cc:648] HybridClock initialized: now 1781967512781114 us; error 68 us; skew 500 ppm
I20260620 14:58:32.781574 7733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:32.784122 7733 webserver.cc:460] Webserver started at http://127.21.159.195:45091/ using document root <none> and password file <none>
I20260620 14:58:32.784852 7733 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:32.784969 7733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:32.785310 7733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:32.788328 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "7007542617d84ac380e166ea6a50fe14"
format_stamp: "Formatted at 2026-06-20 14:58:32 on dist-test-slave-3mqx"
server_key: "dfeaf711ab7a7986d24ed55eed33bc5e"
server_key_iv: "85919ef72473be5dbf1d6bdda755a0b9"
server_key_version: "encryptionkey@0"
I20260620 14:58:32.789146 7733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "7007542617d84ac380e166ea6a50fe14"
format_stamp: "Formatted at 2026-06-20 14:58:32 on dist-test-slave-3mqx"
server_key: "dfeaf711ab7a7986d24ed55eed33bc5e"
server_key_iv: "85919ef72473be5dbf1d6bdda755a0b9"
server_key_version: "encryptionkey@0"
I20260620 14:58:32.794677 7733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 14:58:32.798637 7747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:32.800593 7733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:58:32.800782 7733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "7007542617d84ac380e166ea6a50fe14"
format_stamp: "Formatted at 2026-06-20 14:58:32 on dist-test-slave-3mqx"
server_key: "dfeaf711ab7a7986d24ed55eed33bc5e"
server_key_iv: "85919ef72473be5dbf1d6bdda755a0b9"
server_key_version: "encryptionkey@0"
I20260620 14:58:32.801009 7733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:32.821719 7733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:32.822618 7733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:32.822957 7733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:32.824075 7733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:32.825901 7733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:32.825994 7733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:32.826138 7733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:32.826241 7733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:32.873109 7733 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:32829
I20260620 14:58:32.873107 7859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:32829 every 8 connection(s)
I20260620 14:58:32.875387 7733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:32.882170 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 7733
I20260620 14:58:32.882367 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:32.882949 22143 external_mini_cluster.cc:1428] Setting key f5c0dd3b815053acf864ff74c7199674
I20260620 14:58:32.894608 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.880298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:40143 (local address 127.21.159.254:39767)
0620 14:58:32.880510 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.880518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.881832 (+ 1314us) server_negotiation.cc:407] Connection header received
0620 14:58:32.884645 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.884650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.884740 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.884817 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:32.886734 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.888045 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.890807 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.891634 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.891773 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.893547 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:32.893554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:32.893559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:32.893592 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:32.893677 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:32.893696 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:32.893799 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:32.894108 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:58:32.894222 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:58:32.896165 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.879626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:39767 (local address 127.21.159.195:40143)
0620 14:58:32.881578 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.881637 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.884287 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.885425 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.885437 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.885566 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:32.886469 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.886499 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.888204 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.888212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.890615 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.890634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.891897 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.891908 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.892283 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.893100 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:32.893106 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:32.893181 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:32.893302 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:32.893816 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:32.893830 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:32.893841 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:58:32.894061 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 14:58:32.894561 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":774,"threads_started":1}
I20260620 14:58:32.897862 7860 heartbeater.cc:347] Connected to a master server at 127.21.159.254:39767
I20260620 14:58:32.898242 7860 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:32.899408 7860 heartbeater.cc:511] Master 127.21.159.254:39767 requested a full tablet report, sending...
I20260620 14:58:32.901012 7412 ts_manager.cc:194] Registered new tserver with Master: 7007542617d84ac380e166ea6a50fe14 (127.21.159.195:32829)
I20260620 14:58:32.902372 7412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.195:40143
I20260620 14:58:32.903697 7599 heartbeater.cc:503] Master 127.21.159.254:39767 was elected leader, sending a full tablet report...
I20260620 14:58:32.911909 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:32.923919 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.917077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.21.159.254:39767)
0620 14:58:32.917341 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.917346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.917798 (+ 452us) server_negotiation.cc:407] Connection header received
0620 14:58:32.917925 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.917929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.918007 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.918178 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:32.918765 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.919770 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.921322 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.921772 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.921880 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.923176 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:32.923188 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:32.923194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:32.923226 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:32.923266 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:32.923283 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:32.923385 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:32.923529 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:58:32.923627 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 14:58:32.928676 7412 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:32.936885 7412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:32.974961 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.951328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39364)
0620 14:58:32.965154 (+ 13826us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.965236 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.965500 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.966740 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.966749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.966810 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:32.967506 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.967520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.969355 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.969359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.971686 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.971700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.973880 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.973887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.974101 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.974163 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 14:58:32.974298 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:58:32.974496 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13423,"mutex_wait_us":331,"spinlock_wait_cycles":1280,"thread_start_us":13917,"threads_started":1}
I20260620 14:58:32.975033 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.951998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.21.159.193:46379)
0620 14:58:32.964311 (+ 12313us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.964325 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.965410 (+ 1085us) server_negotiation.cc:407] Connection header received
0620 14:58:32.965826 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.965839 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.966159 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.966400 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:32.967746 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.969186 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.971931 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.973673 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.973899 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.974156 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:32.974402 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:58:32.974625 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12058,"thread_start_us":11888,"threads_started":1}
I20260620 14:58:32.975770 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.952585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.21.159.194:34813)
0620 14:58:32.963678 (+ 11093us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.963694 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.965905 (+ 2211us) server_negotiation.cc:407] Connection header received
0620 14:58:32.966007 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.966012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.966206 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.966451 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:32.967539 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.969297 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.972526 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.974103 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.974253 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.974502 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:32.975185 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 14:58:32.975364 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10813,"thread_start_us":10461,"threads_started":1}
I20260620 14:58:32.976454 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.966033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.21.159.195:32829)
0620 14:58:32.966736 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:32.966744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:32.967725 (+ 981us) server_negotiation.cc:407] Connection header received
0620 14:58:32.968055 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:32.968064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:32.968333 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:32.968533 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:32.969556 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.971534 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.973347 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.974566 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.974769 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.974967 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:32.975914 (+ 947us) server_negotiation.cc:299] Negotiation successful
0620 14:58:32.976098 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":329,"threads_started":1}
I20260620 14:58:32.977016 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.952146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:34813 (local address 127.0.0.1:39676)
0620 14:58:32.965559 (+ 13413us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.965574 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.965732 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.966734 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.966742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.966790 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:32.967257 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.967278 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.969495 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.969504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.972264 (+ 2760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.972279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.974698 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.974705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.974916 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.974964 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:58:32.976206 (+ 1242us) client_negotiation.cc:240] Negotiation successful
0620 14:58:32.976367 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13297,"mutex_wait_us":34,"spinlock_wait_cycles":3840,"thread_start_us":12386,"threads_started":1}
I20260620 14:58:32.977133 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:32.965752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:32829 (local address 127.0.0.1:35814)
0620 14:58:32.967601 (+ 1849us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:32.967614 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:32.967833 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:32.968849 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:32.968855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:32.968889 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:32.969328 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.969342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.971703 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.971708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.973154 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:32.973165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:32.975123 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:32.975133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:32.975704 (+ 571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:32.975743 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:58:32.976707 (+ 964us) client_negotiation.cc:240] Negotiation successful
0620 14:58:32.976874 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1704,"thread_start_us":1518,"threads_started":1}
I20260620 14:58:32.979872 7534 tablet_service.cc:1467] Processing CreateTablet for tablet cc9bd1b929c143cc8718ebcf34576e31 (DEFAULT_TABLE table=test-table [id=9a6228f83cb74f4b80d3ca37670ef4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:32.981290 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9bd1b929c143cc8718ebcf34576e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:32.980958 7795 tablet_service.cc:1467] Processing CreateTablet for tablet cc9bd1b929c143cc8718ebcf34576e31 (DEFAULT_TABLE table=test-table [id=9a6228f83cb74f4b80d3ca37670ef4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:32.980958 7665 tablet_service.cc:1467] Processing CreateTablet for tablet cc9bd1b929c143cc8718ebcf34576e31 (DEFAULT_TABLE table=test-table [id=9a6228f83cb74f4b80d3ca37670ef4d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:32.982321 7665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9bd1b929c143cc8718ebcf34576e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:32.982321 7795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9bd1b929c143cc8718ebcf34576e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:32.994165 7878 tablet_bootstrap.cc:492] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Bootstrap starting.
I20260620 14:58:32.998183 7878 tablet_bootstrap.cc:654] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:32.998111 7879 tablet_bootstrap.cc:492] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: Bootstrap starting.
I20260620 14:58:32.999701 7878 log.cc:824] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:33.001282 7880 tablet_bootstrap.cc:492] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Bootstrap starting.
I20260620 14:58:33.002277 7879 tablet_bootstrap.cc:654] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:33.003542 7878 tablet_bootstrap.cc:492] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: No bootstrap required, opened a new log
I20260620 14:58:33.003665 7879 log.cc:824] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:33.003931 7878 ts_tablet_manager.cc:1397] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 14:58:33.006395 7880 tablet_bootstrap.cc:654] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:33.007326 7879 tablet_bootstrap.cc:492] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: No bootstrap required, opened a new log
I20260620 14:58:33.007629 7879 ts_tablet_manager.cc:1397] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 14:58:33.007781 7880 log.cc:824] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:33.011307 7880 tablet_bootstrap.cc:492] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: No bootstrap required, opened a new log
I20260620 14:58:33.011610 7880 ts_tablet_manager.cc:1397] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 14:58:33.013782 7878 raft_consensus.cc:348] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.014339 7878 raft_consensus.cc:374] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:33.014448 7878 raft_consensus.cc:729] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7007542617d84ac380e166ea6a50fe14, State: Initialized, Role: FOLLOWER
I20260620 14:58:33.015244 7878 consensus_queue.cc:260] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.018179 7879 raft_consensus.cc:348] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.018558 7860 heartbeater.cc:503] Master 127.21.159.254:39767 was elected leader, sending a full tablet report...
I20260620 14:58:33.018710 7879 raft_consensus.cc:374] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:33.018812 7879 raft_consensus.cc:729] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33316cc0e99d4080b368feecdee35679, State: Initialized, Role: FOLLOWER
I20260620 14:58:33.019537 7879 consensus_queue.cc:260] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.020335 7878 ts_tablet_manager.cc:1428] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260620 14:58:33.023690 7880 raft_consensus.cc:348] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.024206 7880 raft_consensus.cc:374] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:33.024318 7880 raft_consensus.cc:729] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738895d02214465962e1ee5b82cc0f2, State: Initialized, Role: FOLLOWER
I20260620 14:58:33.024642 7879 ts_tablet_manager.cc:1428] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260620 14:58:33.025089 7880 consensus_queue.cc:260] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.032292 7730 heartbeater.cc:503] Master 127.21.159.254:39767 was elected leader, sending a full tablet report...
I20260620 14:58:33.033427 7880 ts_tablet_manager.cc:1428] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
W20260620 14:58:33.127794 7861 tablet.cc:2369] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:33.131464 7600 tablet.cc:2369] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:33.131987 7731 tablet.cc:2369] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:33.216820 7886 raft_consensus.cc:482] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:33.217110 7886 raft_consensus.cc:504] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.218861 7886 leader_election.cc:288] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33316cc0e99d4080b368feecdee35679 (127.21.159.193:46379), 7007542617d84ac380e166ea6a50fe14 (127.21.159.195:32829)
I20260620 14:58:33.227308 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.220408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:37781 (local address 127.21.159.195:32829)
0620 14:58:33.220672 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.220678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.221084 (+ 406us) server_negotiation.cc:407] Connection header received
0620 14:58:33.221271 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.221278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.221352 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.221453 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:33.221909 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.222785 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.225254 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.226347 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.226508 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.226587 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:33.226885 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 14:58:33.227022 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:58:33.227326 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.219381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:42553 (local address 127.21.159.193:46379)
0620 14:58:33.219576 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.219587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.220070 (+ 483us) server_negotiation.cc:407] Connection header received
0620 14:58:33.220293 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.220300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.220402 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.220519 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:33.221342 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.222336 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.225189 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.226168 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.226349 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.226405 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:33.226856 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 14:58:33.227009 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:58:33.227455 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.219270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.21.159.194:42553)
0620 14:58:33.219955 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.219977 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:33.220129 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:33.220720 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:33.220725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:33.220760 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:33.221182 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.221198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.222465 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.222468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.224864 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.224880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.226395 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.226404 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.226607 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.226651 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 14:58:33.226788 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:58:33.227023 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":106,"thread_start_us":362,"threads_started":1}
I20260620 14:58:33.227835 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.220130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:32829 (local address 127.21.159.194:37781)
0620 14:58:33.220969 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.220987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:33.221124 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:33.221470 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:33.221473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:33.221493 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:33.221766 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.221773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.222924 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.222930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.225059 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:33.225070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.226542 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.226547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:33.226692 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.226718 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:58:33.226844 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:58:33.227012 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":23,"thread_start_us":500,"threads_started":1}
I20260620 14:58:33.228617 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9bd1b929c143cc8718ebcf34576e31" candidate_uuid: "f738895d02214465962e1ee5b82cc0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33316cc0e99d4080b368feecdee35679" is_pre_election: true
I20260620 14:58:33.228922 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9bd1b929c143cc8718ebcf34576e31" candidate_uuid: "f738895d02214465962e1ee5b82cc0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7007542617d84ac380e166ea6a50fe14" is_pre_election: true
I20260620 14:58:33.229233 7554 raft_consensus.cc:2454] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738895d02214465962e1ee5b82cc0f2 in term 0.
I20260620 14:58:33.229471 7815 raft_consensus.cc:2454] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738895d02214465962e1ee5b82cc0f2 in term 0.
I20260620 14:58:33.229959 7620 leader_election.cc:302] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33316cc0e99d4080b368feecdee35679, f738895d02214465962e1ee5b82cc0f2; no voters:
I20260620 14:58:33.230402 7886 raft_consensus.cc:2781] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:33.230600 7886 raft_consensus.cc:482] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:33.230721 7886 raft_consensus.cc:3037] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:33.233057 7886 raft_consensus.cc:504] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.233659 7886 leader_election.cc:288] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [CANDIDATE]: Term 1 election: Requested vote from peers 33316cc0e99d4080b368feecdee35679 (127.21.159.193:46379), 7007542617d84ac380e166ea6a50fe14 (127.21.159.195:32829)
I20260620 14:58:33.234268 7554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9bd1b929c143cc8718ebcf34576e31" candidate_uuid: "f738895d02214465962e1ee5b82cc0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33316cc0e99d4080b368feecdee35679"
I20260620 14:58:33.234300 7815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9bd1b929c143cc8718ebcf34576e31" candidate_uuid: "f738895d02214465962e1ee5b82cc0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7007542617d84ac380e166ea6a50fe14"
I20260620 14:58:33.234488 7554 raft_consensus.cc:3037] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:33.234512 7815 raft_consensus.cc:3037] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:33.236773 7554 raft_consensus.cc:2454] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f738895d02214465962e1ee5b82cc0f2 in term 1.
I20260620 14:58:33.236819 7815 raft_consensus.cc:2454] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f738895d02214465962e1ee5b82cc0f2 in term 1.
I20260620 14:58:33.237200 7620 leader_election.cc:302] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33316cc0e99d4080b368feecdee35679, f738895d02214465962e1ee5b82cc0f2; no voters:
I20260620 14:58:33.237495 7886 raft_consensus.cc:2781] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:33.237955 7886 raft_consensus.cc:686] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 1 LEADER]: Becoming Leader. State: Replica: f738895d02214465962e1ee5b82cc0f2, State: Running, Role: LEADER
I20260620 14:58:33.238490 7886 consensus_queue.cc:237] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } }
I20260620 14:58:33.243664 7413 catalog_manager.cc:5495] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f738895d02214465962e1ee5b82cc0f2 (127.21.159.194). New cstate: current_term: 1 leader_uuid: "f738895d02214465962e1ee5b82cc0f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f738895d02214465962e1ee5b82cc0f2" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 34813 } health_report { overall_health: HEALTHY } } }
I20260620 14:58:33.262672 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.256177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.21.159.194:34813)
0620 14:58:33.256398 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.256407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.256494 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:58:33.256789 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.256794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.256905 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.257192 (+ 287us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:33.257627 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.258398 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.260242 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.260729 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.260886 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.261445 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:33.261996 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:33.262189 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:33.262255 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:58:33.262347 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 14:58:33.271652 7554 raft_consensus.cc:1261] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 1 FOLLOWER]: Refusing update from remote peer f738895d02214465962e1ee5b82cc0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:33.271652 7815 raft_consensus.cc:1261] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 1 FOLLOWER]: Refusing update from remote peer f738895d02214465962e1ee5b82cc0f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:33.272686 7889 consensus_queue.cc:1035] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7007542617d84ac380e166ea6a50fe14" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:33.273090 7886 consensus_queue.cc:1035] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33316cc0e99d4080b368feecdee35679" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:33.304171 7413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:33.304476 7413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:33.308835 7413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a5c08521418468bb144aa0cd06be5cc: Sending DeleteTablet for 3 replicas of tablet cc9bd1b929c143cc8718ebcf34576e31
I20260620 14:58:33.310204 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9bd1b929c143cc8718ebcf34576e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:33 UTC) from {username='slave'} at 127.0.0.1:39364
I20260620 14:58:33.310441 7665 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9bd1b929c143cc8718ebcf34576e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:33 UTC) from {username='slave'} at 127.0.0.1:39676
I20260620 14:58:33.311415 7795 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9bd1b929c143cc8718ebcf34576e31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:33 UTC) from {username='slave'} at 127.0.0.1:35814
I20260620 14:58:33.314323 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 7472
I20260620 14:58:33.314546 7904 tablet_replica.cc:330] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: stopping tablet replica
I20260620 14:58:33.315071 7904 raft_consensus.cc:2229] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:33.315862 7904 raft_consensus.cc:2258] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:33.323277 7905 tablet_replica.cc:330] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: stopping tablet replica
I20260620 14:58:33.323925 7905 raft_consensus.cc:2229] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:33.324590 7905 raft_consensus.cc:2258] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:33.327512 7905 ts_tablet_manager.cc:1905] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:33.328639 7904 ts_tablet_manager.cc:1905] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:33.333423 7903 tablet_replica.cc:330] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: stopping tablet replica
I20260620 14:58:33.333845 7903 raft_consensus.cc:2229] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:33.334301 7903 raft_consensus.cc:2258] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:33.334704 7905 ts_tablet_manager.cc:1918] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:33.334877 7905 log.cc:1196] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/cc9bd1b929c143cc8718ebcf34576e31
I20260620 14:58:33.335407 7905 ts_tablet_manager.cc:1939] T cc9bd1b929c143cc8718ebcf34576e31 P 7007542617d84ac380e166ea6a50fe14: Deleting consensus metadata
I20260620 14:58:33.336097 7904 ts_tablet_manager.cc:1918] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:33.336282 7904 log.cc:1196] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/cc9bd1b929c143cc8718ebcf34576e31
I20260620 14:58:33.336822 7904 ts_tablet_manager.cc:1939] T cc9bd1b929c143cc8718ebcf34576e31 P f738895d02214465962e1ee5b82cc0f2: Deleting consensus metadata
I20260620 14:58:33.336901 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.315849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.21.159.193:46379)
0620 14:58:33.316039 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.316045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.322385 (+ 6340us) server_negotiation.cc:407] Connection header received
0620 14:58:33.325171 (+ 2786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.325181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.325370 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.325472 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:33.326448 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.327916 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.330595 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.331586 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.331712 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.335125 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:33.335139 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:33.335155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:33.335965 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:33.336027 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:33.336052 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:33.336141 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:33.336424 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:58:33.336551 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:58:33.337280 7903 ts_tablet_manager.cc:1905] T cc9bd1b929c143cc8718ebcf34576e31 P 33316cc0e99d4080b368feecdee35679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:33.337473 7399 catalog_manager.cc:4841] TS 7007542617d84ac380e166ea6a50fe14 (127.21.159.195:32829): tablet cc9bd1b929c143cc8718ebcf34576e31 (table test-table [id=9a6228f83cb74f4b80d3ca37670ef4d5]) successfully deleted
I20260620 14:58:33.338531 7594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:33.338536 7397 catalog_manager.cc:4841] TS f738895d02214465962e1ee5b82cc0f2 (127.21.159.194:34813): tablet cc9bd1b929c143cc8718ebcf34576e31 (table test-table [id=9a6228f83cb74f4b80d3ca37670ef4d5]) successfully deleted
I20260620 14:58:33.450873 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 7472
W20260620 14:58:33.474915 7399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.159.193:46379 (error 108)
I20260620 14:58:33.475760 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 7603
I20260620 14:58:33.477339 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.476837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39382)
0620 14:58:33.477019 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.477160 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 14:58:33.477584 7399 catalog_manager.cc:4584] TS 33316cc0e99d4080b368feecdee35679 (127.21.159.193:46379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc9bd1b929c143cc8718ebcf34576e31: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
I20260620 14:58:33.484498 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.476556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.21.159.194:34813)
0620 14:58:33.476733 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.476744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.476766 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:58:33.477026 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.477033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.477214 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.477322 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:33.477935 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.478901 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.480936 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.481454 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.481572 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.482785 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:33.482794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:33.482809 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:33.483602 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:33.483652 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:33.483674 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:33.483762 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:33.483949 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:58:33.484037 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:58:33.485340 7725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:33.531417 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.530704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39390)
0620 14:58:33.530982 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.531220 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 14:58:33.532578 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.532179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39392)
0620 14:58:33.532353 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.532448 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 14:58:33.593438 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.592886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39404)
0620 14:58:33.593158 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.593255 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 14:58:33.594650 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.594270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39406)
0620 14:58:33.594431 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.594516 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:58:33.603138 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 7603
I20260620 14:58:33.623378 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 7733
I20260620 14:58:33.630707 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.624240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.21.159.195:32829)
0620 14:58:33.624481 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:33.624486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:33.624506 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:58:33.624662 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:33.624667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:33.624766 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:33.624855 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:33.625249 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.625973 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.627455 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:33.627869 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:33.627970 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:33.629143 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:33.629155 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:33.629177 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:33.629847 (+ 670us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:33.629909 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:33.629932 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:33.630038 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:33.630170 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:58:33.630271 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 14:58:33.631605 7855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:33.675577 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.674966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39416)
0620 14:58:33.675290 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.675386 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 14:58:33.676767 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.676317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39426)
0620 14:58:33.676548 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.676660 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260620 14:58:33.722283 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 7733
I20260620 14:58:33.739218 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 7382
I20260620 14:58:33.740566 7443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:33.854315 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:33.852434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:46379 (local address 127.0.0.1:39442)
0620 14:58:33.853535 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:33.854005 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.193:46379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":928}
I20260620 14:58:33.854589 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 7382
2026-06-20T14:58:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3087 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:33 dist-test-slave-3mqx krb5kdc[7914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:33 dist-test-slave-3mqx krb5kdc[7914](info): set up 2 sockets
Jun 20 14:58:33 dist-test-slave-3mqx krb5kdc[7914](info): commencing operation
krb5kdc: starting...
W20260620 14:58:35.019699 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:35 dist-test-slave-3mqx krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:58:35.292042 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:44969
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:44969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:35.644982 7927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:35.645320 7927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:35.645478 7927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:35.652968 7927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:35.653126 7927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:35.653226 7927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:35.653314 7927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:35.653357 7927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:35.663648 7927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:44969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:35.665095 7927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:35.666460 7927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:35.679593 7932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:35.679593 7933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:35.679528 7935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:35.685029 7927 server_base.cc:1034] running on GCE node
I20260620 14:58:35.686069 7927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:35.687884 7927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:35.689174 7927 hybrid_clock.cc:648] HybridClock initialized: now 1781967515689139 us; error 87 us; skew 500 ppm
I20260620 14:58:35.689576 7927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:35 dist-test-slave-3mqx krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967515, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:35.693223 7927 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:58:35.694851 7927 webserver.cc:460] Webserver started at http://127.21.159.254:35563/ using document root <none> and password file <none>
I20260620 14:58:35.695653 7927 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:35.695760 7927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:35.696028 7927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:35.699254 7927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "6535bf28705b4b38a3944a99682d79ee"
format_stamp: "Formatted at 2026-06-20 14:58:35 on dist-test-slave-3mqx"
server_key: "6d37c70bc673d68ac1637bf40dcd1cf9"
server_key_iv: "5baed4e6c074f12a85bb39d92049a55c"
server_key_version: "encryptionkey@0"
I20260620 14:58:35.700011 7927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "6535bf28705b4b38a3944a99682d79ee"
format_stamp: "Formatted at 2026-06-20 14:58:35 on dist-test-slave-3mqx"
server_key: "6d37c70bc673d68ac1637bf40dcd1cf9"
server_key_iv: "5baed4e6c074f12a85bb39d92049a55c"
server_key_version: "encryptionkey@0"
I20260620 14:58:35.705561 7927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:58:35.709408 7942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:35.711004 7927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:58:35.711364 7927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "6535bf28705b4b38a3944a99682d79ee"
format_stamp: "Formatted at 2026-06-20 14:58:35 on dist-test-slave-3mqx"
server_key: "6d37c70bc673d68ac1637bf40dcd1cf9"
server_key_iv: "5baed4e6c074f12a85bb39d92049a55c"
server_key_version: "encryptionkey@0"
I20260620 14:58:35.711603 7927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:35.751705 7927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:35.756666 7927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:35.756992 7927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:35.777112 7927 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:44969
I20260620 14:58:35.777159 7993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:44969 every 8 connection(s)
I20260620 14:58:35.778558 7927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:35.778975 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 7927
I20260620 14:58:35.779210 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:35.779919 22143 external_mini_cluster.cc:1428] Setting key 471ded21ec59fca0eb4951de27e736d3
I20260620 14:58:35.783326 7994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:58:35 dist-test-slave-3mqx krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:35.797291 7994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee: Bootstrap starting.
I20260620 14:58:35.800701 7994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:35.801801 7994 log.cc:824] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:35.804895 7994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee: No bootstrap required, opened a new log
I20260620 14:58:35.805441 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:35.781954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.21.159.254:44969)
0620 14:58:35.782790 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:35.782802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:35.782846 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:58:35.784795 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:35.784818 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:35.785199 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:35.785517 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:35.787406 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.788901 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.790876 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:35.791759 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:35.791960 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:35.797072 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:35.797080 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:35.797098 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:35.797144 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:35.800410 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:35.801327 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:35.801347 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:35.801354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:35.801527 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:35.801948 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:35.801956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:35.801960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:35.802570 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:35.802934 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:35.803937 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0620 14:58:35.804292 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":380,"threads_started":1}
I20260620 14:58:35.813845 7994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6535bf28705b4b38a3944a99682d79ee" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44969 } }
I20260620 14:58:35.814363 7994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:35.814483 7994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6535bf28705b4b38a3944a99682d79ee, State: Initialized, Role: FOLLOWER
I20260620 14:58:35.815261 7994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6535bf28705b4b38a3944a99682d79ee" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44969 } }
I20260620 14:58:35.815543 7994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:35.815708 7994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:35.815902 7994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:35.818207 7994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6535bf28705b4b38a3944a99682d79ee" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44969 } }
I20260620 14:58:35.818768 7994 leader_election.cc:302] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6535bf28705b4b38a3944a99682d79ee; no voters:
I20260620 14:58:35.819558 7994 leader_election.cc:288] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:35.819684 7999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:35.820358 7999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [term 1 LEADER]: Becoming Leader. State: Replica: 6535bf28705b4b38a3944a99682d79ee, State: Running, Role: LEADER
I20260620 14:58:35.821029 7999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6535bf28705b4b38a3944a99682d79ee" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44969 } }
I20260620 14:58:35.822160 7994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:35.824098 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6535bf28705b4b38a3944a99682d79ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6535bf28705b4b38a3944a99682d79ee" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44969 } } }
I20260620 14:58:35.825409 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:35.825955 8001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6535bf28705b4b38a3944a99682d79ee. Latest consensus state: current_term: 1 leader_uuid: "6535bf28705b4b38a3944a99682d79ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6535bf28705b4b38a3944a99682d79ee" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44969 } } }
I20260620 14:58:35.826418 8001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:35.828373 8005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:35.833853 8005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:35.845121 8005 catalog_manager.cc:1347] Generated new cluster ID: 0dc9529e9530492095d984dc7b0d1457
I20260620 14:58:35.845290 8005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:35.867522 8005 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:35.868923 8005 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:35.881228 8005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee: Generated new TSK 0
I20260620 14:58:35.882234 8005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:58:35.947160 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44969
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:36.294688 8020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:36.294978 8020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:36.295132 8020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:36.302589 8020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:36.302711 8020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:36.302825 8020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:58:36.313669 8020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:36.314937 8020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:36.316484 8020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:36.331039 8026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:36.330981 8028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:36.331586 8025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:36.332093 8020 server_base.cc:1034] running on GCE node
I20260620 14:58:36.332710 8020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:36.333693 8020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:36.334903 8020 hybrid_clock.cc:648] HybridClock initialized: now 1781967516334835 us; error 66 us; skew 500 ppm
I20260620 14:58:36.335278 8020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:36 dist-test-slave-3mqx krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967516, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:36.338798 8020 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:58:36.340601 8020 webserver.cc:460] Webserver started at http://127.21.159.193:44213/ using document root <none> and password file <none>
I20260620 14:58:36.341349 8020 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:36.341481 8020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:36.341910 8020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:36.345047 8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "f0cef1e34ca2489fb09dd71da49fe9ea"
format_stamp: "Formatted at 2026-06-20 14:58:36 on dist-test-slave-3mqx"
server_key: "fa4dcbeb7f58ed8c99c068c919d38d5f"
server_key_iv: "abeafdd23b9362ce600f9e4ae6a06434"
server_key_version: "encryptionkey@0"
I20260620 14:58:36.345824 8020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "f0cef1e34ca2489fb09dd71da49fe9ea"
format_stamp: "Formatted at 2026-06-20 14:58:36 on dist-test-slave-3mqx"
server_key: "fa4dcbeb7f58ed8c99c068c919d38d5f"
server_key_iv: "abeafdd23b9362ce600f9e4ae6a06434"
server_key_version: "encryptionkey@0"
I20260620 14:58:36.351608 8020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:58:36.355458 8035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:36.356993 8020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:36.357200 8020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "f0cef1e34ca2489fb09dd71da49fe9ea"
format_stamp: "Formatted at 2026-06-20 14:58:36 on dist-test-slave-3mqx"
server_key: "fa4dcbeb7f58ed8c99c068c919d38d5f"
server_key_iv: "abeafdd23b9362ce600f9e4ae6a06434"
server_key_version: "encryptionkey@0"
I20260620 14:58:36.357439 8020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:36.376879 8020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:36.381825 8020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:36.382155 8020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:36.383278 8020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:36.384850 8020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:36.384959 8020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:36.385154 8020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:36.385236 8020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:36.423010 8020 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:45017
I20260620 14:58:36.423079 8147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:45017 every 8 connection(s)
I20260620 14:58:36.424430 8020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:36.433175 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 8020
I20260620 14:58:36.433395 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:36.433831 22143 external_mini_cluster.cc:1428] Setting key d067e1c15572c7a6b3ea42e333f9a775
Jun 20 14:58:36 dist-test-slave-3mqx krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967516, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:36.449138 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:36.427021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:34577 (local address 127.21.159.254:44969)
0620 14:58:36.427565 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:36.427572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:36.428783 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 14:58:36.431261 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:36.431266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:36.431339 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:36.431432 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:36.433184 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.434099 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.436228 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.436649 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.436753 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.441435 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:36.441445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:36.441453 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:36.441514 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:36.445239 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:36.446399 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:36.446405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:36.446407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:36.446549 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:36.447165 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:36.447169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:36.447171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:36.447455 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:36.447589 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:36.448624 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 14:58:36.448833 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":235,"threads_started":1}
I20260620 14:58:36.450629 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:36.427793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:44969 (local address 127.21.159.193:34577)
0620 14:58:36.428581 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:36.428630 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:36.430862 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:36.431684 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:36.431692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:36.432247 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:36.432954 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.432983 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.434273 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.434280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.436060 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.436076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.436923 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.436932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.437279 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.438168 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:36.438212 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:36.441153 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:36.445437 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:36.445451 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:36.445475 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:36.446222 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:36.446677 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:36.446685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:36.446691 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:36.446982 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:36.447611 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:36.447622 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:36.448120 (+ 498us) client_negotiation.cc:769] Sending connection context
0620 14:58:36.448435 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 14:58:36.449075 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":327,"threads_started":1}
I20260620 14:58:36.452432 8148 heartbeater.cc:347] Connected to a master server at 127.21.159.254:44969
I20260620 14:58:36.452786 8148 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:36.453671 8148 heartbeater.cc:511] Master 127.21.159.254:44969 requested a full tablet report, sending...
I20260620 14:58:36.456099 7959 ts_manager.cc:194] Registered new tserver with Master: f0cef1e34ca2489fb09dd71da49fe9ea (127.21.159.193:45017)
I20260620 14:58:36.457650 7959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:34577
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:58:36.495395 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44969
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:36.848423 8154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:36.848726 8154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:36.848898 8154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:36.856503 8154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:36.856629 8154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:36.856844 8154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:58:36.866941 8154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:36.868427 8154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:36.869865 8154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:36.884220 8159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:36.884195 8162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:36.884243 8160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:36.885229 8154 server_base.cc:1034] running on GCE node
I20260620 14:58:36.885881 8154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:36.887008 8154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:36.888276 8154 hybrid_clock.cc:648] HybridClock initialized: now 1781967516888211 us; error 57 us; skew 500 ppm
I20260620 14:58:36.888612 8154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:36 dist-test-slave-3mqx krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967516, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:36.891917 8154 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:58:36.893568 8154 webserver.cc:460] Webserver started at http://127.21.159.194:40361/ using document root <none> and password file <none>
I20260620 14:58:36.894282 8154 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:36.894438 8154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:36.894766 8154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:36.897816 8154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "4e56a27605c34a4eb45c3dd7d4affead"
format_stamp: "Formatted at 2026-06-20 14:58:36 on dist-test-slave-3mqx"
server_key: "55dee02496bd11b8b7060305d8990b98"
server_key_iv: "0c48afd1e1caa9bcc6bde6568676b05a"
server_key_version: "encryptionkey@0"
I20260620 14:58:36.898622 8154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "4e56a27605c34a4eb45c3dd7d4affead"
format_stamp: "Formatted at 2026-06-20 14:58:36 on dist-test-slave-3mqx"
server_key: "55dee02496bd11b8b7060305d8990b98"
server_key_iv: "0c48afd1e1caa9bcc6bde6568676b05a"
server_key_version: "encryptionkey@0"
I20260620 14:58:36.904175 8154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 14:58:36.908339 8169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:36.909855 8154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:36.910089 8154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "4e56a27605c34a4eb45c3dd7d4affead"
format_stamp: "Formatted at 2026-06-20 14:58:36 on dist-test-slave-3mqx"
server_key: "55dee02496bd11b8b7060305d8990b98"
server_key_iv: "0c48afd1e1caa9bcc6bde6568676b05a"
server_key_version: "encryptionkey@0"
I20260620 14:58:36.910339 8154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:36.937795 8154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:36.942701 8154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:36.943033 8154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:36.944144 8154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:36.945559 8154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:36.945672 8154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:36.945833 8154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:36.945933 8154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:36.980916 8154 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:43509
I20260620 14:58:36.980976 8281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:43509 every 8 connection(s)
I20260620 14:58:36.982276 8154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:36.992367 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 8154
I20260620 14:58:36.992666 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:36.993201 22143 external_mini_cluster.cc:1428] Setting key 7ff4ca0ebc973b929d2c292ff2b321b2
Jun 20 14:58:36 dist-test-slave-3mqx krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967516, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:37.004446 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:36.985081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:56783 (local address 127.21.159.254:44969)
0620 14:58:36.985623 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:36.985630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:36.986586 (+ 956us) server_negotiation.cc:407] Connection header received
0620 14:58:36.988753 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:36.988758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:36.988845 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:36.988961 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:36.990619 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.991482 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.993237 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.993731 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.993872 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.997575 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:36.997580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:36.997583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:36.997617 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:37.000661 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.001785 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.001798 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.001801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.001972 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.002504 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.002510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.002513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.002813 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:37.003003 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.003794 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.004056 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":259,"threads_started":1}
I20260620 14:58:37.005537 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:36.985597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:44969 (local address 127.21.159.194:56783)
0620 14:58:36.986375 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:36.986429 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:36.988433 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:36.989193 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:36.989202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:36.989806 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:36.990416 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.990435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.991644 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.991648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.993085 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:36.993098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:36.993980 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:36.993987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:36.994224 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:36.995014 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:36.995076 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:36.997321 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:37.000839 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:37.000849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:37.000871 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:37.001605 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:37.002138 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:37.002143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:37.002147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:37.002360 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:37.002952 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:37.002961 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:37.003411 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 14:58:37.003715 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 14:58:37.004329 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":298,"threads_started":1}
I20260620 14:58:37.007285 8282 heartbeater.cc:347] Connected to a master server at 127.21.159.254:44969
I20260620 14:58:37.007700 8282 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:37.008677 8282 heartbeater.cc:511] Master 127.21.159.254:44969 requested a full tablet report, sending...
I20260620 14:58:37.010493 7959 ts_manager.cc:194] Registered new tserver with Master: 4e56a27605c34a4eb45c3dd7d4affead (127.21.159.194:43509)
I20260620 14:58:37.011412 7959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:56783
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:58:37.052775 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44969
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:37.394558 8288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:37.394860 8288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:37.394959 8288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:37.402429 8288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:37.402556 8288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:37.402673 8288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:58:37.412853 8288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:37.414137 8288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:37.415566 8288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:37.431387 8296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:37.431496 8293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:37.431460 8294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:37.432331 8288 server_base.cc:1034] running on GCE node
I20260620 14:58:37.432929 8288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:37.433889 8288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:37.435328 8288 hybrid_clock.cc:648] HybridClock initialized: now 1781967517435268 us; error 47 us; skew 500 ppm
I20260620 14:58:37.435655 8288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:37 dist-test-slave-3mqx krb5kdc[7914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:37.439285 8288 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:58:37.440980 8288 webserver.cc:460] Webserver started at http://127.21.159.195:41353/ using document root <none> and password file <none>
I20260620 14:58:37.441745 8288 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:37.441893 8288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:37.442226 8288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:37.445344 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "178791fc14d8462492157ffd6a80a821"
format_stamp: "Formatted at 2026-06-20 14:58:37 on dist-test-slave-3mqx"
server_key: "8259c2d4adc3b7f3ca0e1561076836f5"
server_key_iv: "d7f7354db0b07a44d020a90f9bf4a4dc"
server_key_version: "encryptionkey@0"
I20260620 14:58:37.446197 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "178791fc14d8462492157ffd6a80a821"
format_stamp: "Formatted at 2026-06-20 14:58:37 on dist-test-slave-3mqx"
server_key: "8259c2d4adc3b7f3ca0e1561076836f5"
server_key_iv: "d7f7354db0b07a44d020a90f9bf4a4dc"
server_key_version: "encryptionkey@0"
I20260620 14:58:37.451900 8288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:58:37.455665 8303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:37.457149 8288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:37.457345 8288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "178791fc14d8462492157ffd6a80a821"
format_stamp: "Formatted at 2026-06-20 14:58:37 on dist-test-slave-3mqx"
server_key: "8259c2d4adc3b7f3ca0e1561076836f5"
server_key_iv: "d7f7354db0b07a44d020a90f9bf4a4dc"
server_key_version: "encryptionkey@0"
I20260620 14:58:37.457576 8288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:37.461414 8148 heartbeater.cc:503] Master 127.21.159.254:44969 was elected leader, sending a full tablet report...
I20260620 14:58:37.489365 8288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:37.494091 8288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:37.494469 8288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:37.495649 8288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:37.497052 8288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:37.497174 8288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:37.497347 8288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:37.497406 8288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:37.532907 8288 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:35061
I20260620 14:58:37.533006 8415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:35061 every 8 connection(s)
I20260620 14:58:37.534349 8288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:37.539391 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 8288
I20260620 14:58:37.539619 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:37.540197 22143 external_mini_cluster.cc:1428] Setting key a873e8fe87e99dd9e0243f4b2d421cdf
Jun 20 14:58:37 dist-test-slave-3mqx krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967517, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:37.558391 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.537400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:54367 (local address 127.21.159.254:44969)
0620 14:58:37.537920 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.537928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.539429 (+ 1501us) server_negotiation.cc:407] Connection header received
0620 14:58:37.541700 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.541706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.541822 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.541934 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:37.544102 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.544870 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.546538 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.547168 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.547304 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.551391 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:37.551405 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:37.551409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:37.551455 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:37.554571 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.555704 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.555709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.555711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.555905 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.556523 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.556529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.556533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.556917 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:37.557070 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.557913 (+ 843us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.558114 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":223,"threads_started":1}
I20260620 14:58:37.559639 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.538240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:44969 (local address 127.21.159.195:54367)
0620 14:58:37.539216 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.539284 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.541324 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.542259 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.542269 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.542961 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:37.543881 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.543904 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.545021 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.545026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.546350 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.546362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.547434 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.547440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.547700 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.548462 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:37.548505 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:37.551138 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:37.554759 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:37.554769 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:37.554789 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:37.555556 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:37.556103 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:37.556108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:37.556112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:37.556373 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:37.557107 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:37.557114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:37.557536 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 14:58:37.557802 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 14:58:37.558269 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":402,"threads_started":1}
I20260620 14:58:37.561388 8416 heartbeater.cc:347] Connected to a master server at 127.21.159.254:44969
I20260620 14:58:37.561671 8416 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:37.562469 8416 heartbeater.cc:511] Master 127.21.159.254:44969 requested a full tablet report, sending...
I20260620 14:58:37.564097 7959 ts_manager.cc:194] Registered new tserver with Master: 178791fc14d8462492157ffd6a80a821 (127.21.159.195:35061)
I20260620 14:58:37.564986 7959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:54367
I20260620 14:58:37.569289 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:37.589089 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.573735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.21.159.254:44969)
0620 14:58:37.574051 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.574057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.574318 (+ 261us) server_negotiation.cc:407] Connection header received
0620 14:58:37.574548 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.574553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.574618 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.574728 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:37.576199 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.576915 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.578329 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.578775 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.578924 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.582197 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:37.582204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:37.582210 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:37.582258 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:37.585603 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.586579 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.586585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.586595 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.586741 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.587356 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.587367 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.587370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.587734 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:37.588062 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.588536 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.588758 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:58:37.593719 7959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:37.600876 7959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:37.625818 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.614287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.21.159.193:45017)
0620 14:58:37.615218 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.615233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.615983 (+ 750us) server_negotiation.cc:407] Connection header received
0620 14:58:37.616440 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.616448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.618056 (+ 1608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.618247 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:37.619421 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.620761 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.623344 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.624344 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.624489 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.624610 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.625278 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.625458 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":526,"threads_started":1}
I20260620 14:58:37.625888 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.613515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:45017 (local address 127.0.0.1:39798)
0620 14:58:37.615846 (+ 2331us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.615885 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.616149 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.618674 (+ 2525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.618684 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.618711 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:37.619220 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.619234 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.620953 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.620960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.623169 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.623185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.624742 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.624746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.624940 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.624995 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:58:37.625170 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 14:58:37.625391 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2100,"mutex_wait_us":61,"thread_start_us":2671,"threads_started":1}
I20260620 14:58:37.626609 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.613870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:43509 (local address 127.0.0.1:36942)
0620 14:58:37.616447 (+ 2577us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.616464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.616558 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.617674 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.617683 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.617712 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:37.618101 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.618112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.620050 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.620055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.623029 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.623044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.625320 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.625326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.625470 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.625503 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:58:37.625605 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 14:58:37.625731 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2464,"mutex_wait_us":93,"thread_start_us":1568,"threads_started":1}
I20260620 14:58:37.627000 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.614337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:35061 (local address 127.0.0.1:45578)
0620 14:58:37.617658 (+ 3321us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.617675 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.617791 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.618391 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.618395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.618420 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:37.618930 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.618944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.620668 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.620673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.623152 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.623166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.626037 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.626049 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.626239 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.626276 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:58:37.626352 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:58:37.626482 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3196,"mutex_wait_us":107,"thread_start_us":3044,"threads_started":1}
I20260620 14:58:37.627326 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.614977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.21.159.195:35061)
0620 14:58:37.617228 (+ 2251us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.617237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.617871 (+ 634us) server_negotiation.cc:407] Connection header received
0620 14:58:37.617970 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.617977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.618198 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.618368 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:37.619172 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.620489 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.623311 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.624746 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.624942 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.625083 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.626716 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.626899 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1993,"thread_start_us":1826,"threads_started":1}
I20260620 14:58:37.627364 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.615417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.21.159.194:43509)
0620 14:58:37.616860 (+ 1443us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.616868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.616896 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:58:37.617004 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.617013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.617315 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.617590 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:37.618271 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.619900 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.623261 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.624752 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.624910 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.625073 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.626451 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.626643 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1285,"thread_start_us":1128,"threads_started":1}
I20260620 14:58:37.630856 8083 tablet_service.cc:1467] Processing CreateTablet for tablet fa29bc5afeae438bb6d366477e0f8f35 (DEFAULT_TABLE table=test-table [id=e60238f8d6984cfc855e491b7d4399c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:37.631209 8217 tablet_service.cc:1467] Processing CreateTablet for tablet fa29bc5afeae438bb6d366477e0f8f35 (DEFAULT_TABLE table=test-table [id=e60238f8d6984cfc855e491b7d4399c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:37.631455 8351 tablet_service.cc:1467] Processing CreateTablet for tablet fa29bc5afeae438bb6d366477e0f8f35 (DEFAULT_TABLE table=test-table [id=e60238f8d6984cfc855e491b7d4399c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:37.632663 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa29bc5afeae438bb6d366477e0f8f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:37.632668 8083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa29bc5afeae438bb6d366477e0f8f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:37.632846 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa29bc5afeae438bb6d366477e0f8f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:37.647353 8435 tablet_bootstrap.cc:492] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Bootstrap starting.
I20260620 14:58:37.649921 8437 tablet_bootstrap.cc:492] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Bootstrap starting.
I20260620 14:58:37.651831 8435 tablet_bootstrap.cc:654] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:37.653214 8435 log.cc:824] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:37.653733 8437 tablet_bootstrap.cc:654] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:37.653746 8436 tablet_bootstrap.cc:492] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Bootstrap starting.
I20260620 14:58:37.655130 8437 log.cc:824] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:37.657632 8435 tablet_bootstrap.cc:492] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: No bootstrap required, opened a new log
I20260620 14:58:37.657950 8435 ts_tablet_manager.cc:1397] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 14:58:37.658044 8436 tablet_bootstrap.cc:654] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:37.658411 8437 tablet_bootstrap.cc:492] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: No bootstrap required, opened a new log
I20260620 14:58:37.658676 8437 ts_tablet_manager.cc:1397] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 14:58:37.659340 8436 log.cc:824] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:37.662530 8436 tablet_bootstrap.cc:492] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: No bootstrap required, opened a new log
I20260620 14:58:37.662845 8436 ts_tablet_manager.cc:1397] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 14:58:37.667339 8435 raft_consensus.cc:348] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.667747 8435 raft_consensus.cc:374] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:37.667874 8435 raft_consensus.cc:729] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e56a27605c34a4eb45c3dd7d4affead, State: Initialized, Role: FOLLOWER
I20260620 14:58:37.668505 8435 consensus_queue.cc:260] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.669251 8437 raft_consensus.cc:348] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.669656 8282 heartbeater.cc:503] Master 127.21.159.254:44969 was elected leader, sending a full tablet report...
I20260620 14:58:37.669672 8437 raft_consensus.cc:374] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:37.669771 8437 raft_consensus.cc:729] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0cef1e34ca2489fb09dd71da49fe9ea, State: Initialized, Role: FOLLOWER
I20260620 14:58:37.670364 8435 ts_tablet_manager.cc:1428] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 14:58:37.670389 8437 consensus_queue.cc:260] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.676970 8437 ts_tablet_manager.cc:1428] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
W20260620 14:58:37.677006 8149 tablet.cc:2369] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:37.678637 8436 raft_consensus.cc:348] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.679286 8436 raft_consensus.cc:374] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:37.679440 8436 raft_consensus.cc:729] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178791fc14d8462492157ffd6a80a821, State: Initialized, Role: FOLLOWER
I20260620 14:58:37.680223 8436 consensus_queue.cc:260] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.681442 8416 heartbeater.cc:503] Master 127.21.159.254:44969 was elected leader, sending a full tablet report...
I20260620 14:58:37.682119 8436 ts_tablet_manager.cc:1428] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Time spent starting tablet: real 0.019s user 0.012s sys 0.002s
I20260620 14:58:37.688746 8442 raft_consensus.cc:482] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:37.689061 8442 raft_consensus.cc:504] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.690833 8442 leader_election.cc:288] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e56a27605c34a4eb45c3dd7d4affead (127.21.159.194:43509), 178791fc14d8462492157ffd6a80a821 (127.21.159.195:35061)
I20260620 14:58:37.697513 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.691320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:33217 (local address 127.21.159.194:43509)
0620 14:58:37.691536 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.691544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.691834 (+ 290us) server_negotiation.cc:407] Connection header received
0620 14:58:37.692134 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.692140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.692211 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.692309 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:37.692849 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.693564 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.695474 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.696567 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.696742 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.696819 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.697116 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.697241 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:58:37.697721 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.691011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:43509 (local address 127.21.159.193:33217)
0620 14:58:37.691708 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.691727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.691963 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.692327 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.692332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.692356 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:37.692718 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.692728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.693693 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.693700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.695340 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.695356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.696787 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.696793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.696955 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.696978 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:58:37.697088 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:58:37.697228 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":1,"thread_start_us":504,"threads_started":1}
I20260620 14:58:37.698017 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.691724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:52321 (local address 127.21.159.195:35061)
0620 14:58:37.691941 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.691953 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.692412 (+ 459us) server_negotiation.cc:407] Connection header received
0620 14:58:37.692654 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.692658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.692713 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.692812 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:37.693333 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.694026 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.696065 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.697044 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.697228 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.697304 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.697640 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.697773 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:58:37.698092 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.691329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:35061 (local address 127.21.159.193:52321)
0620 14:58:37.692306 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:37.692321 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:37.692493 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:37.692825 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:37.692830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:37.692852 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:37.693186 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.693194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.694176 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.694181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.695895 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:37.695906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.697270 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.697277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:37.697482 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.697508 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 14:58:37.697615 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:58:37.697753 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":636,"threads_started":1}
I20260620 14:58:37.698873 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29bc5afeae438bb6d366477e0f8f35" candidate_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e56a27605c34a4eb45c3dd7d4affead" is_pre_election: true
I20260620 14:58:37.699237 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29bc5afeae438bb6d366477e0f8f35" candidate_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178791fc14d8462492157ffd6a80a821" is_pre_election: true
I20260620 14:58:37.699522 8237 raft_consensus.cc:2454] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0cef1e34ca2489fb09dd71da49fe9ea in term 0.
I20260620 14:58:37.699792 8371 raft_consensus.cc:2454] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0cef1e34ca2489fb09dd71da49fe9ea in term 0.
I20260620 14:58:37.700316 8037 leader_election.cc:302] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e56a27605c34a4eb45c3dd7d4affead, f0cef1e34ca2489fb09dd71da49fe9ea; no voters:
I20260620 14:58:37.700747 8442 raft_consensus.cc:2781] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:37.700894 8442 raft_consensus.cc:482] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:37.700999 8442 raft_consensus.cc:3037] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:37.703231 8442 raft_consensus.cc:504] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.703894 8442 leader_election.cc:288] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [CANDIDATE]: Term 1 election: Requested vote from peers 4e56a27605c34a4eb45c3dd7d4affead (127.21.159.194:43509), 178791fc14d8462492157ffd6a80a821 (127.21.159.195:35061)
I20260620 14:58:37.704329 8237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29bc5afeae438bb6d366477e0f8f35" candidate_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e56a27605c34a4eb45c3dd7d4affead"
I20260620 14:58:37.704515 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa29bc5afeae438bb6d366477e0f8f35" candidate_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178791fc14d8462492157ffd6a80a821"
I20260620 14:58:37.704547 8237 raft_consensus.cc:3037] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:37.704749 8371 raft_consensus.cc:3037] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:37.706853 8237 raft_consensus.cc:2454] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0cef1e34ca2489fb09dd71da49fe9ea in term 1.
I20260620 14:58:37.707041 8371 raft_consensus.cc:2454] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0cef1e34ca2489fb09dd71da49fe9ea in term 1.
I20260620 14:58:37.707315 8037 leader_election.cc:302] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e56a27605c34a4eb45c3dd7d4affead, f0cef1e34ca2489fb09dd71da49fe9ea; no voters:
I20260620 14:58:37.707631 8442 raft_consensus.cc:2781] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:37.708076 8442 raft_consensus.cc:686] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 1 LEADER]: Becoming Leader. State: Replica: f0cef1e34ca2489fb09dd71da49fe9ea, State: Running, Role: LEADER
I20260620 14:58:37.708631 8442 consensus_queue.cc:237] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } }
I20260620 14:58:37.713358 7959 catalog_manager.cc:5495] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea reported cstate change: term changed from 0 to 1, leader changed from <none> to f0cef1e34ca2489fb09dd71da49fe9ea (127.21.159.193). New cstate: current_term: 1 leader_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0cef1e34ca2489fb09dd71da49fe9ea" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 } health_report { overall_health: UNKNOWN } } }
W20260620 14:58:37.734447 8283 tablet.cc:2369] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:37.754891 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.748976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.21.159.193:45017)
0620 14:58:37.749207 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.749212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.749231 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:58:37.749400 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.749404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.749483 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.749617 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:37.750139 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.750850 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.752538 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.752989 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.753118 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.753648 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:37.754200 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:37.754331 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.754493 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.754570 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 14:58:37.764191 8371 raft_consensus.cc:1261] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 1 FOLLOWER]: Refusing update from remote peer f0cef1e34ca2489fb09dd71da49fe9ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:37.764191 8237 raft_consensus.cc:1261] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 1 FOLLOWER]: Refusing update from remote peer f0cef1e34ca2489fb09dd71da49fe9ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:37.765554 8442 consensus_queue.cc:1035] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "178791fc14d8462492157ffd6a80a821" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:37.765926 8446 consensus_queue.cc:1035] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e56a27605c34a4eb45c3dd7d4affead" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:37.780601 8449 mvcc.cc:205] Tried to move back new op lower bound from 7298938952749768704 to 7298938952538574848. Current Snapshot: MvccSnapshot[applied={T|T < 7298938952749768704}]
I20260620 14:58:37.782141 8451 mvcc.cc:205] Tried to move back new op lower bound from 7298938952749768704 to 7298938952538574848. Current Snapshot: MvccSnapshot[applied={T|T < 7298938952749768704}]
I20260620 14:58:37.812959 7959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:37.814795 7959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:37.820273 7959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6535bf28705b4b38a3944a99682d79ee: Sending DeleteTablet for 3 replicas of tablet fa29bc5afeae438bb6d366477e0f8f35
I20260620 14:58:37.821732 8083 tablet_service.cc:1514] Processing DeleteTablet for tablet fa29bc5afeae438bb6d366477e0f8f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:37 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:39798
I20260620 14:58:37.821764 8217 tablet_service.cc:1514] Processing DeleteTablet for tablet fa29bc5afeae438bb6d366477e0f8f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:37 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:36942
I20260620 14:58:37.822714 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet fa29bc5afeae438bb6d366477e0f8f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:37 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:45578
I20260620 14:58:37.825747 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 8020
I20260620 14:58:37.834329 8461 tablet_replica.cc:330] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: stopping tablet replica
I20260620 14:58:37.834903 8461 raft_consensus.cc:2229] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:37.835518 8461 raft_consensus.cc:2258] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:37.835557 8460 tablet_replica.cc:330] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: stopping tablet replica
I20260620 14:58:37.835935 8460 raft_consensus.cc:2229] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:37.836411 8459 tablet_replica.cc:330] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: stopping tablet replica
I20260620 14:58:37.836599 8460 raft_consensus.cc:2258] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:37.836820 8459 raft_consensus.cc:2229] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:37.837582 8459 raft_consensus.cc:2258] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:37.839174 8460 ts_tablet_manager.cc:1905] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:37.839176 8461 ts_tablet_manager.cc:1905] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:37.840243 8459 ts_tablet_manager.cc:1905] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:37.846307 8461 ts_tablet_manager.cc:1918] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:37.846544 8461 log.cc:1196] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/fa29bc5afeae438bb6d366477e0f8f35
I20260620 14:58:37.846766 8460 ts_tablet_manager.cc:1918] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:37.846971 8460 log.cc:1196] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/fa29bc5afeae438bb6d366477e0f8f35
I20260620 14:58:37.847193 8461 ts_tablet_manager.cc:1939] T fa29bc5afeae438bb6d366477e0f8f35 P 178791fc14d8462492157ffd6a80a821: Deleting consensus metadata
I20260620 14:58:37.847540 8460 ts_tablet_manager.cc:1939] T fa29bc5afeae438bb6d366477e0f8f35 P 4e56a27605c34a4eb45c3dd7d4affead: Deleting consensus metadata
Jun 20 14:58:37 dist-test-slave-3mqx krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:58:37.849403 7945 catalog_manager.cc:4841] TS 178791fc14d8462492157ffd6a80a821 (127.21.159.195:35061): tablet fa29bc5afeae438bb6d366477e0f8f35 (table test-table [id=e60238f8d6984cfc855e491b7d4399c3]) successfully deleted
I20260620 14:58:37.849772 7944 catalog_manager.cc:4841] TS 4e56a27605c34a4eb45c3dd7d4affead (127.21.159.194:43509): tablet fa29bc5afeae438bb6d366477e0f8f35 (table test-table [id=e60238f8d6984cfc855e491b7d4399c3]) successfully deleted
I20260620 14:58:37.853374 8459 ts_tablet_manager.cc:1918] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:37.853588 8459 log.cc:1196] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/fa29bc5afeae438bb6d366477e0f8f35
I20260620 14:58:37.854167 8459 ts_tablet_manager.cc:1939] T fa29bc5afeae438bb6d366477e0f8f35 P f0cef1e34ca2489fb09dd71da49fe9ea: Deleting consensus metadata
I20260620 14:58:37.855974 7943 catalog_manager.cc:4841] TS f0cef1e34ca2489fb09dd71da49fe9ea (127.21.159.193:45017): tablet fa29bc5afeae438bb6d366477e0f8f35 (table test-table [id=e60238f8d6984cfc855e491b7d4399c3]) successfully deleted
I20260620 14:58:37.858721 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.829138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.21.159.193:45017)
0620 14:58:37.829384 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.829394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.832910 (+ 3516us) server_negotiation.cc:407] Connection header received
0620 14:58:37.833119 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.833128 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.833257 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.833397 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:37.837027 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.838332 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.840291 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.840949 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.841065 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.850546 (+ 9481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:37.850561 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:37.850580 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:37.850639 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:37.855261 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.856145 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.856156 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.856159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.856315 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.856757 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.856911 (+ 154us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.856914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.857377 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:37.857796 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:37.858133 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 14:58:37.858377 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:58:37.859539 8143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:37.961673 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 8020
I20260620 14:58:37.980269 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 8154
Jun 20 14:58:37 dist-test-slave-3mqx krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:58:38.000847 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:37.981151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.21.159.194:43509)
0620 14:58:37.981361 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:37.981365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:37.981449 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:58:37.981751 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:37.981756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:37.981825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:37.981918 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:37.983927 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.984829 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.987231 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:37.987690 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:37.987805 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:37.993661 (+ 5856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:37.993679 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:37.993702 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:37.993764 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:37.997452 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.998311 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.998326 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.998329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.998477 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:37.998917 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:37.998923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:37.998926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:37.999461 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:37.999830 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:38.000184 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 14:58:38.000447 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 14:58:38.001619 8277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:38.095006 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 8154
I20260620 14:58:38.113011 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 8288
Jun 20 14:58:38 dist-test-slave-3mqx krb5kdc[7914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:58:38.130112 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:38.114131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.21.159.195:35061)
0620 14:58:38.114329 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:38.114334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:38.114352 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:58:38.114408 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:38.114412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:38.114529 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:38.114621 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:38.116323 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.117223 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.118627 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.119169 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.119275 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.124107 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:38.124125 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:38.124141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:38.124182 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:38.127257 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:38.128022 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:38.128032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:38.128036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:38.128190 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:38.128590 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:38.128595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:38.128597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:38.128992 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:38.129264 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:38.129496 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:58:38.129683 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:58:38.130779 8411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:38.224926 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 8288
I20260620 14:58:38.241822 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 7927
I20260620 14:58:38.243047 7989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:38.358434 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 7927
2026-06-20T14:58:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4512 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:38Z Disabled control of system clock
I20260620 14:58:38.442483 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46667
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:46667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:58:38.786791 8470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:38.787286 8470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:38.787438 8470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:38.795123 8470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:38.795259 8470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:38.795342 8470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:38.795425 8470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:38.795487 8470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:38.805222 8470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:46667
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:38.806658 8470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:38.808585 8470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:38.821143 8478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:38.821228 8476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:38.821185 8475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:38.829123 8470 server_base.cc:1034] running on GCE node
I20260620 14:58:38.830049 8470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:38.831637 8470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:38.832909 8470 hybrid_clock.cc:648] HybridClock initialized: now 1781967518832845 us; error 76 us; skew 500 ppm
I20260620 14:58:38.833308 8470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:38.835830 8470 webserver.cc:460] Webserver started at http://127.21.159.254:43753/ using document root <none> and password file <none>
I20260620 14:58:38.836598 8470 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:38.836743 8470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:38.837080 8470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:38.840173 8470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "f7ce319eb66b43bb8ddbb59e692cbc29"
format_stamp: "Formatted at 2026-06-20 14:58:38 on dist-test-slave-3mqx"
server_key: "ab117964126f8fbfc18165d73c791aa0"
server_key_iv: "00a9a9e93ae23db65b1c07d8aed96c17"
server_key_version: "encryptionkey@0"
I20260620 14:58:38.841024 8470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "f7ce319eb66b43bb8ddbb59e692cbc29"
format_stamp: "Formatted at 2026-06-20 14:58:38 on dist-test-slave-3mqx"
server_key: "ab117964126f8fbfc18165d73c791aa0"
server_key_iv: "00a9a9e93ae23db65b1c07d8aed96c17"
server_key_version: "encryptionkey@0"
I20260620 14:58:38.846458 8470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:38.850337 8484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:38.852000 8470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:38.852217 8470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "f7ce319eb66b43bb8ddbb59e692cbc29"
format_stamp: "Formatted at 2026-06-20 14:58:38 on dist-test-slave-3mqx"
server_key: "ab117964126f8fbfc18165d73c791aa0"
server_key_iv: "00a9a9e93ae23db65b1c07d8aed96c17"
server_key_version: "encryptionkey@0"
I20260620 14:58:38.852483 8470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:38.908239 8470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:38.909266 8470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:38.909590 8470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:38.929358 8470 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:46667
I20260620 14:58:38.929378 8535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:46667 every 8 connection(s)
I20260620 14:58:38.930867 8470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:38.934863 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:38.940168 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 8470
I20260620 14:58:38.940376 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:38.940915 22143 external_mini_cluster.cc:1428] Setting key 813b534e3845a595ebab4ffd1653308a
I20260620 14:58:38.947350 8536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29: Bootstrap starting.
I20260620 14:58:38.950661 8536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:38.951846 8536 log.cc:824] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:38.955042 8536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29: No bootstrap required, opened a new log
I20260620 14:58:38.956105 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:38.943001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.21.159.254:46667)
0620 14:58:38.944459 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:38.944478 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:38.944544 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:58:38.946562 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:38.946587 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:38.947230 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:38.947718 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:38.948315 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.949631 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.951201 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:38.951936 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:38.952162 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:38.953460 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:38.953468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:38.953484 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:38.954066 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:38.954152 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:38.954175 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:38.954313 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:38.954545 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:58:38.954867 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":418,"threads_started":1}
I20260620 14:58:38.963524 8536 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46667 } }
I20260620 14:58:38.963984 8536 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:38.964115 8536 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ce319eb66b43bb8ddbb59e692cbc29, State: Initialized, Role: FOLLOWER
I20260620 14:58:38.964707 8536 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46667 } }
I20260620 14:58:38.964967 8536 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:38.965145 8536 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:38.965348 8536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:38.967792 8536 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46667 } }
I20260620 14:58:38.968329 8536 leader_election.cc:302] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ce319eb66b43bb8ddbb59e692cbc29; no voters:
I20260620 14:58:38.968961 8536 leader_election.cc:288] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:38.969118 8541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:38.970093 8541 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [term 1 LEADER]: Becoming Leader. State: Replica: f7ce319eb66b43bb8ddbb59e692cbc29, State: Running, Role: LEADER
I20260620 14:58:38.970692 8541 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46667 } }
I20260620 14:58:38.971408 8536 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:38.974928 8542 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46667 } } }
I20260620 14:58:38.974952 8543 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ce319eb66b43bb8ddbb59e692cbc29. Latest consensus state: current_term: 1 leader_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ce319eb66b43bb8ddbb59e692cbc29" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46667 } } }
I20260620 14:58:38.975602 8543 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:38.975605 8542 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:38.976836 8549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:38.982550 8549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:38.992920 8549 catalog_manager.cc:1347] Generated new cluster ID: dcc94fe406d448faa00821b9a9abfc33
I20260620 14:58:38.993067 8549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:39.008396 8549 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:39.009912 8549 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:39.021108 8549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29: Generated new TSK 0
I20260620 14:58:39.022156 8549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:39.040759 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46667
--builtin_ntp_servers=127.21.159.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:58:39.408540 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:39.408951 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:39.409135 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:39.417560 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:39.417716 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:39.417893 8560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:58:39.427475 8560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:39.428871 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:39.430732 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:39.444890 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:39.444850 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:39.444824 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:39.445817 8560 server_base.cc:1034] running on GCE node
I20260620 14:58:39.446434 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:39.447503 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:39.448764 8560 hybrid_clock.cc:648] HybridClock initialized: now 1781967519448719 us; error 46 us; skew 500 ppm
I20260620 14:58:39.449146 8560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:39.451953 8560 webserver.cc:460] Webserver started at http://127.21.159.193:45145/ using document root <none> and password file <none>
I20260620 14:58:39.452709 8560 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:39.452872 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:39.453223 8560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:39.456729 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "14caa9e0cf674f04a284ec9bd48fcaee"
format_stamp: "Formatted at 2026-06-20 14:58:39 on dist-test-slave-3mqx"
server_key: "8507842dd20ac31a99ca902a2a86fb65"
server_key_iv: "381f8b381466c75a7566826e6f06572d"
server_key_version: "encryptionkey@0"
I20260620 14:58:39.457676 8560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "14caa9e0cf674f04a284ec9bd48fcaee"
format_stamp: "Formatted at 2026-06-20 14:58:39 on dist-test-slave-3mqx"
server_key: "8507842dd20ac31a99ca902a2a86fb65"
server_key_iv: "381f8b381466c75a7566826e6f06572d"
server_key_version: "encryptionkey@0"
I20260620 14:58:39.463454 8560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:58:39.467487 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:39.469190 8560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:39.469434 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "14caa9e0cf674f04a284ec9bd48fcaee"
format_stamp: "Formatted at 2026-06-20 14:58:39 on dist-test-slave-3mqx"
server_key: "8507842dd20ac31a99ca902a2a86fb65"
server_key_iv: "381f8b381466c75a7566826e6f06572d"
server_key_version: "encryptionkey@0"
I20260620 14:58:39.469725 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:39.514614 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:39.515710 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:39.516134 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:39.517357 8560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:39.519351 8560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:39.519507 8560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:39.519682 8560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:39.519796 8560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:39.560329 8560 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:44149
I20260620 14:58:39.560382 8686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:44149 every 8 connection(s)
I20260620 14:58:39.561966 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:39.562321 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 8560
I20260620 14:58:39.562561 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:39.563364 22143 external_mini_cluster.cc:1428] Setting key af2dae07f820e930b3e0ba0000acd14f
I20260620 14:58:39.567312 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46667
--builtin_ntp_servers=127.21.159.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:58:39.578070 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:39.565332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:41479 (local address 127.21.159.254:46667)
0620 14:58:39.565899 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:39.565906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:39.567427 (+ 1521us) server_negotiation.cc:407] Connection header received
0620 14:58:39.569765 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:39.569772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:39.569877 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:39.570022 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:39.571407 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.572304 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.574375 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.574952 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.575198 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.577106 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:39.577112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:39.577116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:39.577137 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:39.577180 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:39.577193 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:39.577324 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:39.577618 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:58:39.577708 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
I20260620 14:58:39.579800 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:39.566142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46667 (local address 127.21.159.193:41479)
0620 14:58:39.567267 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:39.567306 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:39.569413 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:39.570297 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:39.570307 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:39.570424 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:39.571177 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.571225 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.572509 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.572516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.574201 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:39.574216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:39.575296 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:39.575309 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:39.575729 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:39.576654 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:39.576661 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:39.576726 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:39.576857 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:39.577337 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:39.577353 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:39.577366 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:58:39.577607 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 14:58:39.578121 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":335,"threads_started":1}
I20260620 14:58:39.581576 8687 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46667
I20260620 14:58:39.581903 8687 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:39.582756 8687 heartbeater.cc:511] Master 127.21.159.254:46667 requested a full tablet report, sending...
I20260620 14:58:39.585199 8501 ts_manager.cc:194] Registered new tserver with Master: 14caa9e0cf674f04a284ec9bd48fcaee (127.21.159.193:44149)
I20260620 14:58:39.586757 8501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.193:41479
W20260620 14:58:39.922694 8691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:39.923022 8691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:39.923236 8691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:39.930986 8691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:39.931169 8691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:39.931380 8691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:58:39.941051 8691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:39.942457 8691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:39.944367 8691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:39.958338 8696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:39.958423 8699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:39.958360 8697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:39.959417 8691 server_base.cc:1034] running on GCE node
I20260620 14:58:39.960083 8691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:39.961144 8691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:39.962347 8691 hybrid_clock.cc:648] HybridClock initialized: now 1781967519962279 us; error 61 us; skew 500 ppm
I20260620 14:58:39.962679 8691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:39.965060 8691 webserver.cc:460] Webserver started at http://127.21.159.194:44881/ using document root <none> and password file <none>
I20260620 14:58:39.965790 8691 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:39.965957 8691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:39.966303 8691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:39.969372 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "52c14d095ffb44a99fbc733915e68ee9"
format_stamp: "Formatted at 2026-06-20 14:58:39 on dist-test-slave-3mqx"
server_key: "f431b381880fc15dc51d859ece0183e4"
server_key_iv: "269d7accb263f036bf9c93b92a83bfdd"
server_key_version: "encryptionkey@0"
I20260620 14:58:39.970109 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "52c14d095ffb44a99fbc733915e68ee9"
format_stamp: "Formatted at 2026-06-20 14:58:39 on dist-test-slave-3mqx"
server_key: "f431b381880fc15dc51d859ece0183e4"
server_key_iv: "269d7accb263f036bf9c93b92a83bfdd"
server_key_version: "encryptionkey@0"
I20260620 14:58:39.975543 8691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 14:58:39.979465 8705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:39.980993 8691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:39.981226 8691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "52c14d095ffb44a99fbc733915e68ee9"
format_stamp: "Formatted at 2026-06-20 14:58:39 on dist-test-slave-3mqx"
server_key: "f431b381880fc15dc51d859ece0183e4"
server_key_iv: "269d7accb263f036bf9c93b92a83bfdd"
server_key_version: "encryptionkey@0"
I20260620 14:58:39.981431 8691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:39.998230 8691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:39.999236 8691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:39.999568 8691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:40.000690 8691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:40.002240 8691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:40.002329 8691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:40.002511 8691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:40.002590 8691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:40.038676 8691 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:39395
I20260620 14:58:40.038775 8817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:39395 every 8 connection(s)
I20260620 14:58:40.040158 8691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:40.048066 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 8691
I20260620 14:58:40.048259 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:40.048717 22143 external_mini_cluster.cc:1428] Setting key de1b99aba225eb77ef37afb4e42ba9ce
I20260620 14:58:40.051723 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46667
--builtin_ntp_servers=127.21.159.212:33889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:58:40.054247 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.042888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:56179 (local address 127.21.159.254:46667)
0620 14:58:40.043128 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.043136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.044647 (+ 1511us) server_negotiation.cc:407] Connection header received
0620 14:58:40.046522 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.046528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.046637 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.046760 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:40.047985 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.048920 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.051230 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.051653 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.051797 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.053386 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:40.053391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:40.053394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:40.053414 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:40.053448 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:40.053460 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:40.053561 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.053808 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.053917 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:58:40.055457 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.043538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46667 (local address 127.21.159.194:56179)
0620 14:58:40.044454 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.044497 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.046216 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.047018 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.047028 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.047190 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:40.047799 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.047817 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.049084 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.049094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.050994 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.051007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.051906 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.051915 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.052196 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.052983 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:40.052986 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:40.053032 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:40.053154 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:40.053559 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:40.053574 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:40.053584 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:58:40.053781 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 14:58:40.054250 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":263,"threads_started":1}
I20260620 14:58:40.056998 8818 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46667
I20260620 14:58:40.057344 8818 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:40.058313 8818 heartbeater.cc:511] Master 127.21.159.254:46667 requested a full tablet report, sending...
I20260620 14:58:40.060120 8501 ts_manager.cc:194] Registered new tserver with Master: 52c14d095ffb44a99fbc733915e68ee9 (127.21.159.194:39395)
I20260620 14:58:40.060951 8501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.194:56179
W20260620 14:58:40.409233 8821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:40.409535 8821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:40.409703 8821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:40.417948 8821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:40.418068 8821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:40.418303 8821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:58:40.428745 8821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:33889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:46667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:40.430204 8821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:40.432238 8821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:40.446684 8829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:40.446727 8827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:40.446645 8826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:40.449023 8821 server_base.cc:1034] running on GCE node
I20260620 14:58:40.449678 8821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:40.450752 8821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:40.452020 8821 hybrid_clock.cc:648] HybridClock initialized: now 1781967520451937 us; error 70 us; skew 500 ppm
I20260620 14:58:40.452405 8821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:40.455330 8821 webserver.cc:460] Webserver started at http://127.21.159.195:46337/ using document root <none> and password file <none>
I20260620 14:58:40.456123 8821 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:40.456272 8821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:40.456744 8821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:40.460253 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "94a322566dc6475cbd5bc4bbd425ed33"
format_stamp: "Formatted at 2026-06-20 14:58:40 on dist-test-slave-3mqx"
server_key: "76339901215fb5573b2ce29c104fc283"
server_key_iv: "7f70033d4ee800c80e53b142ce0358b5"
server_key_version: "encryptionkey@0"
I20260620 14:58:40.461135 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "94a322566dc6475cbd5bc4bbd425ed33"
format_stamp: "Formatted at 2026-06-20 14:58:40 on dist-test-slave-3mqx"
server_key: "76339901215fb5573b2ce29c104fc283"
server_key_iv: "7f70033d4ee800c80e53b142ce0358b5"
server_key_version: "encryptionkey@0"
I20260620 14:58:40.467279 8821 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 14:58:40.471604 8835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:40.473393 8821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:58:40.473634 8821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "94a322566dc6475cbd5bc4bbd425ed33"
format_stamp: "Formatted at 2026-06-20 14:58:40 on dist-test-slave-3mqx"
server_key: "76339901215fb5573b2ce29c104fc283"
server_key_iv: "7f70033d4ee800c80e53b142ce0358b5"
server_key_version: "encryptionkey@0"
I20260620 14:58:40.473984 8821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:40.511906 8821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:40.512892 8821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:40.513242 8821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:40.514406 8821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:40.516059 8821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:40.516201 8821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:40.516355 8821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:40.516456 8821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:40.555343 8821 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:43469
I20260620 14:58:40.555409 8947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:43469 every 8 connection(s)
I20260620 14:58:40.556982 8821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:40.560966 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 8821
I20260620 14:58:40.561157 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:40.561748 22143 external_mini_cluster.cc:1428] Setting key 5c19b32b0b759f7d1106c8b63a65e8a9
I20260620 14:58:40.572301 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.560007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:38271 (local address 127.21.159.254:46667)
0620 14:58:40.560505 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.560511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.562025 (+ 1514us) server_negotiation.cc:407] Connection header received
0620 14:58:40.564818 (+ 2793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.564827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.564924 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.565077 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:40.566318 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.567597 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.569287 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.569709 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.569821 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.571419 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:40.571428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:40.571432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:40.571457 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:40.571499 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:40.571512 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:40.571587 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.571855 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.571948 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":234,"threads_started":1}
I20260620 14:58:40.573554 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.560680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46667 (local address 127.21.159.195:38271)
0620 14:58:40.561846 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.561897 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.564479 (+ 2582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.565349 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.565359 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.565463 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:40.566130 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.566148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.567775 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.567780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.569120 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.569137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.569954 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.569960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.570182 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.570993 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:40.570996 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:40.571043 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:40.571217 (+ 174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:40.571609 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:40.571618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:40.571625 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:58:40.571840 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 14:58:40.572260 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":424,"threads_started":1}
I20260620 14:58:40.575408 8948 heartbeater.cc:347] Connected to a master server at 127.21.159.254:46667
I20260620 14:58:40.575765 8948 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:40.576840 8948 heartbeater.cc:511] Master 127.21.159.254:46667 requested a full tablet report, sending...
I20260620 14:58:40.578588 8501 ts_manager.cc:194] Registered new tserver with Master: 94a322566dc6475cbd5bc4bbd425ed33 (127.21.159.195:43469)
I20260620 14:58:40.579797 8501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.195:38271
I20260620 14:58:40.590082 8687 heartbeater.cc:503] Master 127.21.159.254:46667 was elected leader, sending a full tablet report...
I20260620 14:58:40.591706 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:40.602887 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.596461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.21.159.254:46667)
0620 14:58:40.596690 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.596695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.597104 (+ 409us) server_negotiation.cc:407] Connection header received
0620 14:58:40.597306 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.597310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.597379 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.597496 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:40.598047 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.598769 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.600493 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.600921 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.601036 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.602223 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:40.602228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:40.602231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:40.602250 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:40.602281 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:40.602292 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:40.602410 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.602600 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.602646 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 14:58:40.607689 8500 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:40.614961 8500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:40.642820 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.628712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:44149 (local address 127.0.0.1:48270)
0620 14:58:40.629760 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.629801 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.629990 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.632490 (+ 2500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.632500 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.632539 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:40.633067 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.633081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.635557 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.635563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.639207 (+ 3644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.639224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.641619 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.641627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.641846 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.641928 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 14:58:40.642080 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:58:40.642332 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":881,"thread_start_us":560,"threads_started":1}
I20260620 14:58:40.643736 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.628777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:39395 (local address 127.0.0.1:36920)
0620 14:58:40.631475 (+ 2698us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.631496 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.631649 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.632729 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.632740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.632762 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:40.634649 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.634663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.636769 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.636775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.640638 (+ 3863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.640651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.642744 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.642758 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.643031 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.643095 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:58:40.643196 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:58:40.643338 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2534,"mutex_wait_us":30,"thread_start_us":2359,"threads_started":1}
I20260620 14:58:40.643827 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.629314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.21.159.194:39395)
0620 14:58:40.631223 (+ 1909us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.631236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.631801 (+ 565us) server_negotiation.cc:407] Connection header received
0620 14:58:40.631907 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.631916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.632216 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.632432 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:40.634895 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.636583 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.640857 (+ 4274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.642201 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.642363 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.642594 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.643250 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.643439 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1672,"thread_start_us":1538,"threads_started":1}
I20260620 14:58:40.643916 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.628944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.21.159.193:44149)
0620 14:58:40.631762 (+ 2818us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.631773 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.631798 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:58:40.631907 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.631916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.632213 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.633294 (+ 1081us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:40.633597 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.635369 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.639480 (+ 4111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.641401 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.641596 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.641843 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.642871 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.643373 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2609,"thread_start_us":2436,"threads_started":1}
I20260620 14:58:40.648211 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.631646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43469 (local address 127.0.0.1:51762)
0620 14:58:40.637929 (+ 6283us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.637954 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.638094 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.638792 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.638797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.638826 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:40.639391 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.639427 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.642138 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.642142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.644911 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.644924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.647324 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.647331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.647527 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.647565 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:58:40.647661 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:58:40.647803 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6144,"mutex_wait_us":1,"thread_start_us":5919,"threads_started":1}
I20260620 14:58:40.648204 8622 tablet_service.cc:1467] Processing CreateTablet for tablet 97a369e46e1245e1ac83e9d0e5c3ee63 (DEFAULT_TABLE table=test-table [id=6b570bca1d654ecfa148888c733cef10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:40.648183 8753 tablet_service.cc:1467] Processing CreateTablet for tablet 97a369e46e1245e1ac83e9d0e5c3ee63 (DEFAULT_TABLE table=test-table [id=6b570bca1d654ecfa148888c733cef10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:40.648976 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.632207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.21.159.195:43469)
0620 14:58:40.637534 (+ 5327us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.637545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.638099 (+ 554us) server_negotiation.cc:407] Connection header received
0620 14:58:40.638332 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.638341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.638586 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.638811 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:40.639686 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.641310 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.645118 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.646689 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.646907 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.647164 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.648243 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.648558 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5112,"thread_start_us":4947,"threads_started":1}
I20260620 14:58:40.649716 8622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a369e46e1245e1ac83e9d0e5c3ee63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:40.649770 8753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a369e46e1245e1ac83e9d0e5c3ee63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:40.653189 8883 tablet_service.cc:1467] Processing CreateTablet for tablet 97a369e46e1245e1ac83e9d0e5c3ee63 (DEFAULT_TABLE table=test-table [id=6b570bca1d654ecfa148888c733cef10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:40.654968 8883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a369e46e1245e1ac83e9d0e5c3ee63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:40.663502 8967 tablet_bootstrap.cc:492] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Bootstrap starting.
I20260620 14:58:40.667361 8967 tablet_bootstrap.cc:654] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:40.667915 8968 tablet_bootstrap.cc:492] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Bootstrap starting.
I20260620 14:58:40.668905 8967 log.cc:824] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:40.672057 8968 tablet_bootstrap.cc:654] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:40.672230 8970 tablet_bootstrap.cc:492] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Bootstrap starting.
I20260620 14:58:40.673017 8967 tablet_bootstrap.cc:492] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: No bootstrap required, opened a new log
I20260620 14:58:40.673337 8967 ts_tablet_manager.cc:1397] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 14:58:40.673589 8968 log.cc:824] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:40.676281 8970 tablet_bootstrap.cc:654] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:40.677286 8968 tablet_bootstrap.cc:492] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: No bootstrap required, opened a new log
I20260620 14:58:40.677678 8968 ts_tablet_manager.cc:1397] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 14:58:40.677747 8970 log.cc:824] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:40.681348 8970 tablet_bootstrap.cc:492] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: No bootstrap required, opened a new log
I20260620 14:58:40.681715 8970 ts_tablet_manager.cc:1397] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 14:58:40.683115 8967 raft_consensus.cc:348] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.683574 8967 raft_consensus.cc:374] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:40.683691 8967 raft_consensus.cc:729] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52c14d095ffb44a99fbc733915e68ee9, State: Initialized, Role: FOLLOWER
I20260620 14:58:40.684379 8967 consensus_queue.cc:260] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.685599 8818 heartbeater.cc:503] Master 127.21.159.254:46667 was elected leader, sending a full tablet report...
I20260620 14:58:40.686499 8967 ts_tablet_manager.cc:1428] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260620 14:58:40.689708 8968 raft_consensus.cc:348] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.690228 8968 raft_consensus.cc:374] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:40.690341 8968 raft_consensus.cc:729] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14caa9e0cf674f04a284ec9bd48fcaee, State: Initialized, Role: FOLLOWER
I20260620 14:58:40.691035 8968 consensus_queue.cc:260] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.693085 8968 ts_tablet_manager.cc:1428] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260620 14:58:40.694243 8970 raft_consensus.cc:348] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.694664 8970 raft_consensus.cc:374] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:40.694787 8970 raft_consensus.cc:729] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a322566dc6475cbd5bc4bbd425ed33, State: Initialized, Role: FOLLOWER
I20260620 14:58:40.695581 8970 consensus_queue.cc:260] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.696902 8948 heartbeater.cc:503] Master 127.21.159.254:46667 was elected leader, sending a full tablet report...
I20260620 14:58:40.697824 8970 ts_tablet_manager.cc:1428] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 14:58:40.735460 8974 raft_consensus.cc:482] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:40.735877 8974 raft_consensus.cc:504] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.737900 8974 leader_election.cc:288] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52c14d095ffb44a99fbc733915e68ee9 (127.21.159.194:39395), 94a322566dc6475cbd5bc4bbd425ed33 (127.21.159.195:43469)
I20260620 14:58:40.744724 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.738438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:46975 (local address 127.21.159.194:39395)
0620 14:58:40.738653 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.738657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.739322 (+ 665us) server_negotiation.cc:407] Connection header received
0620 14:58:40.739570 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.739580 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.739713 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.739782 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:40.740481 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.741157 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.743008 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.743773 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.743970 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.744042 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.744314 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.744466 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:58:40.744832 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.738380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:39395 (local address 127.21.159.193:46975)
0620 14:58:40.739170 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.739193 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.739338 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.739996 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.739999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.740020 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:40.740307 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.740315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.741303 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.741309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.742799 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.742811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.743979 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.743986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.744148 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.744171 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:58:40.744286 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:58:40.744423 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":4,"thread_start_us":401,"threads_started":1}
I20260620 14:58:40.745390 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.739556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:60209 (local address 127.21.159.195:43469)
0620 14:58:40.739809 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.739815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.740059 (+ 244us) server_negotiation.cc:407] Connection header received
0620 14:58:40.740289 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.740295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.740363 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.740486 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:40.741080 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.741796 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.743654 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.744432 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.744620 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.744687 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.745042 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.745169 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 14:58:40.746035 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.739330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:43469 (local address 127.21.159.193:60209)
0620 14:58:40.739940 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:40.739955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:40.740147 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:40.740486 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:40.740491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:40.740512 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:40.740846 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.740854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.741949 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.741955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.743466 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:40.743478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.744626 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.744633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:40.744832 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.744868 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:58:40.745593 (+ 725us) client_negotiation.cc:240] Negotiation successful
0620 14:58:40.745728 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
I20260620 14:58:40.745929 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a369e46e1245e1ac83e9d0e5c3ee63" candidate_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c14d095ffb44a99fbc733915e68ee9" is_pre_election: true
I20260620 14:58:40.746596 8773 raft_consensus.cc:2454] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14caa9e0cf674f04a284ec9bd48fcaee in term 0.
I20260620 14:58:40.747371 8576 leader_election.cc:302] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14caa9e0cf674f04a284ec9bd48fcaee, 52c14d095ffb44a99fbc733915e68ee9; no voters:
I20260620 14:58:40.747370 8903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a369e46e1245e1ac83e9d0e5c3ee63" candidate_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a322566dc6475cbd5bc4bbd425ed33" is_pre_election: true
I20260620 14:58:40.747797 8974 raft_consensus.cc:2781] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:40.747941 8974 raft_consensus.cc:482] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:40.748065 8974 raft_consensus.cc:3037] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:40.748114 8903 raft_consensus.cc:2454] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14caa9e0cf674f04a284ec9bd48fcaee in term 0.
I20260620 14:58:40.750592 8974 raft_consensus.cc:504] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.751368 8974 leader_election.cc:288] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [CANDIDATE]: Term 1 election: Requested vote from peers 52c14d095ffb44a99fbc733915e68ee9 (127.21.159.194:39395), 94a322566dc6475cbd5bc4bbd425ed33 (127.21.159.195:43469)
I20260620 14:58:40.751950 8773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a369e46e1245e1ac83e9d0e5c3ee63" candidate_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52c14d095ffb44a99fbc733915e68ee9"
I20260620 14:58:40.751978 8903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a369e46e1245e1ac83e9d0e5c3ee63" candidate_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a322566dc6475cbd5bc4bbd425ed33"
I20260620 14:58:40.752194 8773 raft_consensus.cc:3037] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:40.752200 8903 raft_consensus.cc:3037] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:40.754616 8773 raft_consensus.cc:2454] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14caa9e0cf674f04a284ec9bd48fcaee in term 1.
I20260620 14:58:40.754835 8903 raft_consensus.cc:2454] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14caa9e0cf674f04a284ec9bd48fcaee in term 1.
I20260620 14:58:40.755146 8576 leader_election.cc:302] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14caa9e0cf674f04a284ec9bd48fcaee, 52c14d095ffb44a99fbc733915e68ee9; no voters:
I20260620 14:58:40.755486 8974 raft_consensus.cc:2781] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:40.756080 8974 raft_consensus.cc:686] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 1 LEADER]: Becoming Leader. State: Replica: 14caa9e0cf674f04a284ec9bd48fcaee, State: Running, Role: LEADER
I20260620 14:58:40.756654 8974 consensus_queue.cc:237] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } }
I20260620 14:58:40.761723 8500 catalog_manager.cc:5495] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee reported cstate change: term changed from 0 to 1, leader changed from <none> to 14caa9e0cf674f04a284ec9bd48fcaee (127.21.159.193). New cstate: current_term: 1 leader_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14caa9e0cf674f04a284ec9bd48fcaee" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 } health_report { overall_health: UNKNOWN } } }
W20260620 14:58:40.792413 8819 tablet.cc:2369] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:40.801496 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.795334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.21.159.193:44149)
0620 14:58:40.795579 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.795586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.795608 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:58:40.795807 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.795812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.795910 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.796139 (+ 229us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:40.796543 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.797333 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.798903 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.799466 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.799631 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.800305 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:40.800845 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:40.801072 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.801167 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.801232 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 14:58:40.808981 8949 tablet.cc:2369] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:40.810709 8903 raft_consensus.cc:1261] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 1 FOLLOWER]: Refusing update from remote peer 14caa9e0cf674f04a284ec9bd48fcaee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:40.810797 8773 raft_consensus.cc:1261] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 1 FOLLOWER]: Refusing update from remote peer 14caa9e0cf674f04a284ec9bd48fcaee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:40.811853 8978 consensus_queue.cc:1035] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [LEADER]: Connected to new peer: Peer: permanent_uuid: "94a322566dc6475cbd5bc4bbd425ed33" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 43469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:40.812273 8974 consensus_queue.cc:1035] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [LEADER]: Connected to new peer: Peer: permanent_uuid: "52c14d095ffb44a99fbc733915e68ee9" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:58:40.815299 8688 tablet.cc:2369] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:40.839855 8500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:40.840185 8500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:40.844220 8500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7ce319eb66b43bb8ddbb59e692cbc29: Sending DeleteTablet for 3 replicas of tablet 97a369e46e1245e1ac83e9d0e5c3ee63
I20260620 14:58:40.845422 8622 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a369e46e1245e1ac83e9d0e5c3ee63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:40 UTC) from {username='slave'} at 127.0.0.1:48270
I20260620 14:58:40.845945 8753 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a369e46e1245e1ac83e9d0e5c3ee63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:40 UTC) from {username='slave'} at 127.0.0.1:36920
I20260620 14:58:40.846874 8991 tablet_replica.cc:330] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: stopping tablet replica
I20260620 14:58:40.846776 8883 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a369e46e1245e1ac83e9d0e5c3ee63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:40 UTC) from {username='slave'} at 127.0.0.1:51762
I20260620 14:58:40.847393 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 8560
I20260620 14:58:40.857895 8991 raft_consensus.cc:2229] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:40.858279 8993 tablet_replica.cc:330] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: stopping tablet replica
I20260620 14:58:40.858875 8993 raft_consensus.cc:2229] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:40.858911 8991 raft_consensus.cc:2258] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:40.859498 8993 raft_consensus.cc:2258] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:40.862654 8993 ts_tablet_manager.cc:1905] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:40.865007 8992 tablet_replica.cc:330] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: stopping tablet replica
I20260620 14:58:40.865420 8992 raft_consensus.cc:2229] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:40.865900 8992 raft_consensus.cc:2258] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:40.867620 8991 ts_tablet_manager.cc:1905] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:40.870700 8993 ts_tablet_manager.cc:1918] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:40.870884 8993 log.cc:1196] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/97a369e46e1245e1ac83e9d0e5c3ee63
I20260620 14:58:40.871519 8993 ts_tablet_manager.cc:1939] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 94a322566dc6475cbd5bc4bbd425ed33: Deleting consensus metadata
I20260620 14:58:40.873804 8486 catalog_manager.cc:4841] TS 94a322566dc6475cbd5bc4bbd425ed33 (127.21.159.195:43469): tablet 97a369e46e1245e1ac83e9d0e5c3ee63 (table test-table [id=6b570bca1d654ecfa148888c733cef10]) successfully deleted
I20260620 14:58:40.874293 8991 ts_tablet_manager.cc:1918] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:40.874538 8991 log.cc:1196] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/97a369e46e1245e1ac83e9d0e5c3ee63
I20260620 14:58:40.875231 8991 ts_tablet_manager.cc:1939] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 14caa9e0cf674f04a284ec9bd48fcaee: Deleting consensus metadata
I20260620 14:58:40.876293 8992 ts_tablet_manager.cc:1905] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:40.877120 8488 catalog_manager.cc:4841] TS 14caa9e0cf674f04a284ec9bd48fcaee (127.21.159.193:44149): tablet 97a369e46e1245e1ac83e9d0e5c3ee63 (table test-table [id=6b570bca1d654ecfa148888c733cef10]) successfully deleted
I20260620 14:58:40.880681 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:40.850907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.21.159.193:44149)
0620 14:58:40.851240 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:40.851249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:40.863969 (+ 12720us) server_negotiation.cc:407] Connection header received
0620 14:58:40.866356 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:40.866365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:40.866481 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:40.866548 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:40.871154 (+ 4606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.872508 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.875859 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:40.876640 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:40.876758 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:40.878785 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:40.878798 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:40.878818 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:40.879728 (+ 910us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:40.879793 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:40.879832 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:40.879963 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:40.880159 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:58:40.880282 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260620 14:58:40.881476 8682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:40.883384 8992 ts_tablet_manager.cc:1918] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:40.883634 8992 log.cc:1196] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/97a369e46e1245e1ac83e9d0e5c3ee63
I20260620 14:58:40.884241 8992 ts_tablet_manager.cc:1939] T 97a369e46e1245e1ac83e9d0e5c3ee63 P 52c14d095ffb44a99fbc733915e68ee9: Deleting consensus metadata
I20260620 14:58:40.886365 8486 catalog_manager.cc:4841] TS 52c14d095ffb44a99fbc733915e68ee9 (127.21.159.194:39395): tablet 97a369e46e1245e1ac83e9d0e5c3ee63 (table test-table [id=6b570bca1d654ecfa148888c733cef10]) successfully deleted
I20260620 14:58:40.990633 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 8560
I20260620 14:58:41.011442 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 8691
I20260620 14:58:41.019001 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.012283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.21.159.194:39395)
0620 14:58:41.012479 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:41.012483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:41.012511 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:58:41.012578 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:41.012582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:41.012689 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:41.012787 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:41.013207 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.014056 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.015483 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.016202 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.016318 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.017464 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:41.017475 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:41.017498 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:41.018186 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:41.018247 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:41.018271 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:41.018377 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:41.018567 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:58:41.018676 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:58:41.019905 8813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:41.113793 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 8691
I20260620 14:58:41.130790 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 8821
I20260620 14:58:41.138388 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:41.131751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.21.159.195:43469)
0620 14:58:41.131935 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:41.131940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:41.131963 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:58:41.132090 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:41.132094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:41.132184 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:41.132268 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:41.132684 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.133468 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.135005 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:41.135559 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:41.135665 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:41.136737 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:41.136746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:41.136761 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:41.137343 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:41.137388 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:41.137408 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:41.137505 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:41.137851 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 14:58:41.137962 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:58:41.139272 8943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:41.234279 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 8821
I20260620 14:58:41.253475 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 8470
I20260620 14:58:41.254657 8531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:41.367429 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 8470
2026-06-20T14:58:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3005 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:41 dist-test-slave-3mqx krb5kdc[9001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:41 dist-test-slave-3mqx krb5kdc[9001](info): set up 2 sockets
Jun 20 14:58:41 dist-test-slave-3mqx krb5kdc[9001](info): commencing operation
krb5kdc: starting...
W20260620 14:58:42.587684 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:42 dist-test-slave-3mqx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:58:42.850943 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:44263
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:44263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:43.213297 9014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:43.213608 9014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:43.213768 9014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:43.222535 9014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:43.222662 9014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:43.222765 9014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:43.222877 9014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:43.222966 9014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:43.233441 9014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:44263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:43.234860 9014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:43.236295 9014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:43.249092 9019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:43.249189 9020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:43.249084 9022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:43.250810 9014 server_base.cc:1034] running on GCE node
I20260620 14:58:43.251735 9014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:43.253289 9014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:43.254554 9014 hybrid_clock.cc:648] HybridClock initialized: now 1781967523254488 us; error 50 us; skew 500 ppm
I20260620 14:58:43.254937 9014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:43 dist-test-slave-3mqx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:43.258322 9014 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:58:43.260177 9014 webserver.cc:460] Webserver started at http://127.21.159.254:44805/ using document root <none> and password file <none>
I20260620 14:58:43.260957 9014 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:43.261111 9014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:43.261433 9014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:43.264675 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "3ce850ae7db24dadaeb4cf35223c2f6c"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-3mqx"
server_key: "63e48ab926d4e0a728889bb295ce5af1"
server_key_iv: "ff5d244f3c63d881d03140a3e2dce003"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.265466 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "3ce850ae7db24dadaeb4cf35223c2f6c"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-3mqx"
server_key: "63e48ab926d4e0a728889bb295ce5af1"
server_key_iv: "ff5d244f3c63d881d03140a3e2dce003"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.270937 9014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:58:43.274591 9029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:43.276331 9014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:43.276604 9014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "3ce850ae7db24dadaeb4cf35223c2f6c"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-3mqx"
server_key: "63e48ab926d4e0a728889bb295ce5af1"
server_key_iv: "ff5d244f3c63d881d03140a3e2dce003"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.276865 9014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:43.302325 9014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:43.307551 9014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:43.307905 9014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:43.327576 9014 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:44263
I20260620 14:58:43.327585 9080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:44263 every 8 connection(s)
I20260620 14:58:43.329188 9014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:43.333256 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:43.338696 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9014
I20260620 14:58:43.338918 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:43.339409 22143 external_mini_cluster.cc:1428] Setting key 49cea0930cfeca8d02a2b198bfe470db
I20260620 14:58:43.344491 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c: Bootstrap starting.
I20260620 14:58:43.349133 9081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:43.350565 9081 log.cc:824] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:43.354866 9081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c: No bootstrap required, opened a new log
Jun 20 14:58:43 dist-test-slave-3mqx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:43.364137 9081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44263 } }
I20260620 14:58:43.364684 9081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:43.364830 9081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce850ae7db24dadaeb4cf35223c2f6c, State: Initialized, Role: FOLLOWER
I20260620 14:58:43.365568 9081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44263 } }
I20260620 14:58:43.365864 9081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:43.365899 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:43.341707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.21.159.254:44263)
0620 14:58:43.342602 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:43.342614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:43.342658 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:58:43.344447 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:43.344472 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:43.344930 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:43.345257 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:43.347459 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:43.348857 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:43.351004 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:43.352208 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:43.352373 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:43.357788 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:43.357797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:43.357816 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:43.357862 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:43.361245 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:43.362043 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:43.362051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:43.362057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:43.362246 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:43.362694 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:43.362699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:43.362703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:43.363372 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:43.363677 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:43.364135 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 14:58:43.364654 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":420,"threads_started":1}
I20260620 14:58:43.366034 9081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:43.366252 9081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:43.369462 9081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44263 } }
I20260620 14:58:43.369957 9081 leader_election.cc:302] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ce850ae7db24dadaeb4cf35223c2f6c; no voters:
I20260620 14:58:43.370517 9081 leader_election.cc:288] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:43.370676 9086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:43.371476 9086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [term 1 LEADER]: Becoming Leader. State: Replica: 3ce850ae7db24dadaeb4cf35223c2f6c, State: Running, Role: LEADER
I20260620 14:58:43.372054 9086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44263 } }
I20260620 14:58:43.372854 9081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:43.374795 9087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44263 } } }
I20260620 14:58:43.375207 9087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:43.376233 9088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ce850ae7db24dadaeb4cf35223c2f6c. Latest consensus state: current_term: 1 leader_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ce850ae7db24dadaeb4cf35223c2f6c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 44263 } } }
I20260620 14:58:43.376638 9088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:43.380234 9091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:43.387566 9091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:43.399694 9091 catalog_manager.cc:1347] Generated new cluster ID: d0856bcd2b5145dd9e116c1de15bbc76
I20260620 14:58:43.399834 9091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:43.414426 9091 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:43.416364 9091 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:43.431983 9091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c: Generated new TSK 0
I20260620 14:58:43.433086 9091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.159.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
Entry for principal kudu/127.21.159.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab.
I20260620 14:58:43.495038 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44263
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:43.851478 9107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:43.851780 9107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:43.851958 9107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:43.860082 9107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:43.860208 9107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:43.860407 9107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:58:43.870810 9107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.193.keytab
--principal=kudu/127.21.159.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:43.872202 9107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:43.873700 9107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:43.887652 9112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:43.887629 9113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:43.887631 9115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:43.888610 9107 server_base.cc:1034] running on GCE node
I20260620 14:58:43.889228 9107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:43.890165 9107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:43.891386 9107 hybrid_clock.cc:648] HybridClock initialized: now 1781967523891321 us; error 59 us; skew 500 ppm
I20260620 14:58:43.891709 9107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:43 dist-test-slave-3mqx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:43.894894 9107 init.cc:377] Logged in from keytab as kudu/127.21.159.193@KRBTEST.COM (short username kudu)
I20260620 14:58:43.896440 9107 webserver.cc:460] Webserver started at http://127.21.159.193:35283/ using document root <none> and password file <none>
I20260620 14:58:43.897146 9107 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:43.897289 9107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:43.897603 9107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:43.900615 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "5525791d26be4d37aa665cd03cc7cd42"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-3mqx"
server_key: "62eee2fa749300aabe0beb53d0aa9f6e"
server_key_iv: "45a8d4da8e8873995b8048cefbc2959f"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.901396 9107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "5525791d26be4d37aa665cd03cc7cd42"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-3mqx"
server_key: "62eee2fa749300aabe0beb53d0aa9f6e"
server_key_iv: "45a8d4da8e8873995b8048cefbc2959f"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.906767 9107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:43.910348 9122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:43.911761 9107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:58:43.911938 9107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "5525791d26be4d37aa665cd03cc7cd42"
format_stamp: "Formatted at 2026-06-20 14:58:43 on dist-test-slave-3mqx"
server_key: "62eee2fa749300aabe0beb53d0aa9f6e"
server_key_iv: "45a8d4da8e8873995b8048cefbc2959f"
server_key_version: "encryptionkey@0"
I20260620 14:58:43.912147 9107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:43.928575 9107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:43.933403 9107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:43.933718 9107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:43.934762 9107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:43.936196 9107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:43.936298 9107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:43.936444 9107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:43.936522 9107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:43.970770 9107 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:37033
I20260620 14:58:43.970808 9234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:37033 every 8 connection(s)
I20260620 14:58:43.972146 9107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:43.974007 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9107
I20260620 14:58:43.974215 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:43.974665 22143 external_mini_cluster.cc:1428] Setting key 48c4c8d05eb92a809421c179fa80b544
Jun 20 14:58:43 dist-test-slave-3mqx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967523, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.193@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:43.994727 9237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:43.975331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:43451 (local address 127.21.159.254:44263)
0620 14:58:43.975739 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:43.975743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:43.976711 (+ 968us) server_negotiation.cc:407] Connection header received
0620 14:58:43.978630 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:43.978638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:43.978731 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:43.978852 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:43.980489 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:43.981726 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:43.983367 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:43.984147 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:43.984293 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:43.988549 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:43.988555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:43.988558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:43.988593 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:43.991496 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:43.992316 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:43.992321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:43.992323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:43.992449 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:43.992915 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:43.992924 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:43.992926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:43.993197 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:43.993361 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:43.994110 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 14:58:43.994349 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":199,"threads_started":1}
I20260620 14:58:43.995949 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:43.975621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:44263 (local address 127.21.159.193:43451)
0620 14:58:43.976533 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:43.976588 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:43.978348 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:43.979141 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:43.979158 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:43.979720 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:43.980311 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:43.980329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:43.981901 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:43.981905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:43.983230 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:43.983242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:43.984347 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:43.984352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:43.984677 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:43.985442 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:43.985477 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:43.988288 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:43.991667 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:43.991677 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:43.991696 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:43.992170 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:43.992572 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:43.992577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:43.992580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:43.992789 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:43.993333 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:43.993343 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:43.993718 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 14:58:43.994002 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 14:58:43.994587 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":407,"threads_started":1}
I20260620 14:58:43.997748 9235 heartbeater.cc:347] Connected to a master server at 127.21.159.254:44263
I20260620 14:58:43.998067 9235 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:43.998831 9235 heartbeater.cc:511] Master 127.21.159.254:44263 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.21.159.194@KRBTEST.COM; defaulting to no policy
I20260620 14:58:44.001312 9046 ts_manager.cc:194] Registered new tserver with Master: 5525791d26be4d37aa665cd03cc7cd42 (127.21.159.193:37033)
I20260620 14:58:44.002910 9046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.193@KRBTEST.COM'} at 127.21.159.193:43451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
Entry for principal kudu/127.21.159.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab.
I20260620 14:58:44.036288 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44263
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:44.369580 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:44.369875 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:44.370086 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:44.378108 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:44.378286 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:44.378484 9241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:58:44.388943 9241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.194.keytab
--principal=kudu/127.21.159.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:44.390358 9241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:44.392002 9241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:44.407986 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.407914 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.407959 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:44.408913 9241 server_base.cc:1034] running on GCE node
I20260620 14:58:44.409505 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:44.410480 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:44.411723 9241 hybrid_clock.cc:648] HybridClock initialized: now 1781967524411646 us; error 59 us; skew 500 ppm
I20260620 14:58:44.412026 9241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:44 dist-test-slave-3mqx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:44.415326 9241 init.cc:377] Logged in from keytab as kudu/127.21.159.194@KRBTEST.COM (short username kudu)
I20260620 14:58:44.416862 9241 webserver.cc:460] Webserver started at http://127.21.159.194:38357/ using document root <none> and password file <none>
I20260620 14:58:44.417562 9241 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:44.417663 9241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:44.417932 9241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:44.421015 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-3mqx"
server_key: "7d5c25045abeb2bc7002c6d16d4a99aa"
server_key_iv: "e66ee008e2fd0992031ebcd6f5179b31"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.421798 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-3mqx"
server_key: "7d5c25045abeb2bc7002c6d16d4a99aa"
server_key_iv: "e66ee008e2fd0992031ebcd6f5179b31"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.427362 9241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:58:44.431229 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.432699 9241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:44.432896 9241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-3mqx"
server_key: "7d5c25045abeb2bc7002c6d16d4a99aa"
server_key_iv: "e66ee008e2fd0992031ebcd6f5179b31"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.433147 9241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:44.451840 9241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:44.456100 9241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:44.456403 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:44.457407 9241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:44.458777 9241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:44.458851 9241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.459033 9241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:44.459153 9241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 14:58:44.493870 9241 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:44163
I20260620 14:58:44.493916 9368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:44163 every 8 connection(s)
I20260620 14:58:44.495375 9241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:44.501621 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9241
I20260620 14:58:44.501823 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:44.502241 22143 external_mini_cluster.cc:1428] Setting key 57760f2e709498965a28ecfb4760b380
Jun 20 14:58:44 dist-test-slave-3mqx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.194@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:44.521039 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.498972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:44263 (local address 127.21.159.194:59823)
0620 14:58:44.499808 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:44.499870 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:44.501541 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:44.502364 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:44.502377 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:44.503027 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:44.503906 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.503933 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.505385 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.505389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.506711 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:44.506724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.507974 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.507980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:44.508207 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.508967 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:44.508993 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:44.511905 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:44.515949 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.515961 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.515984 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.516713 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.517266 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:44.517271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:44.517274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:44.517547 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:44.518209 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:44.518218 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:44.518688 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 14:58:44.518993 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 14:58:44.519598 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":264,"threads_started":1}
I20260620 14:58:44.522631 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:44.498368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:59823 (local address 127.21.159.254:44263)
0620 14:58:44.498834 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:44.498841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:44.500000 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 14:58:44.501840 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:44.501846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:44.501954 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:44.502081 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:44.504117 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.505200 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.506889 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:44.507714 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:44.507861 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:44.512255 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:44.512264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:44.512272 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:44.512321 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:44.515731 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.516917 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.516938 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.516942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.517128 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:44.517695 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:44.517699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:44.517702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:44.518062 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:44.521819 (+ 3757us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:44.522002 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:58:44.522256 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":225,"threads_started":1}
I20260620 14:58:44.524319 9369 heartbeater.cc:347] Connected to a master server at 127.21.159.254:44263
I20260620 14:58:44.524796 9369 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:44.525795 9369 heartbeater.cc:511] Master 127.21.159.254:44263 requested a full tablet report, sending...
I20260620 14:58:44.527608 9046 ts_manager.cc:194] Registered new tserver with Master: 6c675a1ffb5b46af8f1b7e9d6887b06a (127.21.159.194:44163)
I20260620 14:58:44.528501 9046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.194@KRBTEST.COM'} at 127.21.159.194:59823
WARNING: no policy specified for kudu/127.21.159.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
Entry for principal kudu/127.21.159.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab.
I20260620 14:58:44.566771 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44263
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:44.911471 9375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:44.911842 9375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:44.912032 9375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:44.919606 9375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:44.919729 9375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:44.919935 9375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:58:44.929956 9375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.195.keytab
--principal=kudu/127.21.159.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:44.931475 9375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:44.932920 9375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:44.946879 9380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.946975 9381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:44.946954 9383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:44.947674 9375 server_base.cc:1034] running on GCE node
I20260620 14:58:44.948277 9375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:44.949280 9375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:44.950488 9375 hybrid_clock.cc:648] HybridClock initialized: now 1781967524950446 us; error 32 us; skew 500 ppm
I20260620 14:58:44.950785 9375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:44 dist-test-slave-3mqx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:44.954156 9375 init.cc:377] Logged in from keytab as kudu/127.21.159.195@KRBTEST.COM (short username kudu)
I20260620 14:58:44.955744 9375 webserver.cc:460] Webserver started at http://127.21.159.195:45595/ using document root <none> and password file <none>
I20260620 14:58:44.956436 9375 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:44.956542 9375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:44.956795 9375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:44.959826 9375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "370291040d2449db9565068a5b467f26"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-3mqx"
server_key: "da3f22f163f86a17a7d230a44c3c295f"
server_key_iv: "accf15edc0b46ef6dd6b1e11c641ce0e"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.960539 9375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "370291040d2449db9565068a5b467f26"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-3mqx"
server_key: "da3f22f163f86a17a7d230a44c3c295f"
server_key_iv: "accf15edc0b46ef6dd6b1e11c641ce0e"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.965865 9375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:44.969408 9390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:44.970899 9375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:58:44.971113 9375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "370291040d2449db9565068a5b467f26"
format_stamp: "Formatted at 2026-06-20 14:58:44 on dist-test-slave-3mqx"
server_key: "da3f22f163f86a17a7d230a44c3c295f"
server_key_iv: "accf15edc0b46ef6dd6b1e11c641ce0e"
server_key_version: "encryptionkey@0"
I20260620 14:58:44.971307 9375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:45.000821 9375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:45.005661 9375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:45.005973 9375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:45.007191 9375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:45.007195 9235 heartbeater.cc:503] Master 127.21.159.254:44263 was elected leader, sending a full tablet report...
I20260620 14:58:45.009454 9375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:45.009557 9375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:45.009706 9375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:45.009778 9375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:45.044703 9375 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:46661
I20260620 14:58:45.044760 9502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:46661 every 8 connection(s)
I20260620 14:58:45.046133 9375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:45.054143 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9375
I20260620 14:58:45.054365 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:45.054831 22143 external_mini_cluster.cc:1428] Setting key f01508db49d2403d8df81a8e66160375
Jun 20 14:58:45 dist-test-slave-3mqx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967524, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.195@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:58:45.067386 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.048829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:41199 (local address 127.21.159.254:44263)
0620 14:58:45.049267 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.049271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.050339 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 14:58:45.052274 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.052280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.052387 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.052513 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:45.054230 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.055285 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.057189 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.057761 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.057900 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.061467 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:45.061472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:45.061475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:45.061520 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:45.064550 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.065269 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.065273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.065275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.065400 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.065786 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.065789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.065791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.066097 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:45.066223 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.066872 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.067045 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":201,"threads_started":1}
I20260620 14:58:45.068528 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.049454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:44263 (local address 127.21.159.195:41199)
0620 14:58:45.050180 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:45.050217 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:45.051993 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:45.052758 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:45.052766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:45.053307 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:45.054024 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.054050 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.055476 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.055482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.057019 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.057032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.058005 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.058010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.058223 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.058905 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:58:45.058952 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:45.061271 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:45.064690 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:45.064697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:45.064711 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:45.065161 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:45.065493 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:58:45.065496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:58:45.065498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:58:45.065690 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:58:45.066216 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:45.066222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:45.066564 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 14:58:45.066796 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 14:58:45.067290 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":271,"threads_started":1}
I20260620 14:58:45.070212 9503 heartbeater.cc:347] Connected to a master server at 127.21.159.254:44263
I20260620 14:58:45.070547 9503 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:45.071295 9503 heartbeater.cc:511] Master 127.21.159.254:44263 requested a full tablet report, sending...
I20260620 14:58:45.072628 9046 ts_manager.cc:194] Registered new tserver with Master: 370291040d2449db9565068a5b467f26 (127.21.159.195:46661)
I20260620 14:58:45.073515 9046 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.159.195@KRBTEST.COM'} at 127.21.159.195:41199
I20260620 14:58:45.083016 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:45.100592 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.087247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.21.159.254:44263)
0620 14:58:45.087418 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.087422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.087917 (+ 495us) server_negotiation.cc:407] Connection header received
0620 14:58:45.088083 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.088087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.088152 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.088239 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:45.089713 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.090438 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.091870 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.092270 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.092365 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.094795 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:45.094800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:45.094804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:45.094842 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:45.097571 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.098418 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.098427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.098432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.098573 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.098970 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.098976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.098980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.099393 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:45.099682 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.100043 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.100282 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:58:45.104694 9046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:58:45.111927 9046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:45.154698 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.132406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.21.159.193:37033)
0620 14:58:45.145738 (+ 13332us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.145747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.147493 (+ 1746us) server_negotiation.cc:407] Connection header received
0620 14:58:45.147595 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.147602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.147809 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.148025 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:45.148661 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.149699 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.152244 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.153478 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.153687 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.153881 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.154159 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.154338 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13152,"thread_start_us":13339,"threads_started":1}
I20260620 14:58:45.154742 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.132021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:37033 (local address 127.0.0.1:37826)
0620 14:58:45.147306 (+ 15285us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:45.147329 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:45.147473 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:45.148067 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:45.148071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:45.148106 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:45.148486 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.148494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.149840 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.149848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.152079 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.152090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.153693 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.153700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.153894 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.153952 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 14:58:45.154093 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 14:58:45.154287 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15116,"mutex_wait_us":29,"thread_start_us":14083,"threads_started":1}
I20260620 14:58:45.155804 9517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.128486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:44163 (local address 127.0.0.1:57296)
0620 14:58:45.144765 (+ 16279us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:45.144829 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:45.145125 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:45.147687 (+ 2562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:45.147697 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:45.147733 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:45.148127 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.148143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.150422 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.150427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.152863 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.152879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.154990 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.154995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.155226 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.155257 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:58:45.155336 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:58:45.155477 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15889,"thread_start_us":16895,"threads_started":1}
I20260620 14:58:45.156415 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.128758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:46661 (local address 127.0.0.1:60042)
0620 14:58:45.146489 (+ 17731us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:45.146513 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:45.146661 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:45.148938 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:45.148946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:45.148976 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:45.149311 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.149322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.151239 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.151245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.153304 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.153315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.155614 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.155620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.155784 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.155819 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:58:45.155914 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:58:45.156065 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17528,"thread_start_us":18320,"threads_started":1}
I20260620 14:58:45.156589 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.128500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.21.159.194:44163)
0620 14:58:45.141210 (+ 12710us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.141224 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.144973 (+ 3749us) server_negotiation.cc:407] Connection header received
0620 14:58:45.145432 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.145442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.145762 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.145998 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:45.148568 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.150229 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.153052 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.154493 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.154654 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.154833 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.156076 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.156249 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12536,"thread_start_us":12404,"threads_started":1}
I20260620 14:58:45.157274 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.128924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.21.159.195:46661)
0620 14:58:45.148082 (+ 19158us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.148092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.148129 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:58:45.148227 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.148237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.148465 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.148650 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:45.149506 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.151046 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.153488 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.155080 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.155277 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.155453 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.156835 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.157013 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18944,"thread_start_us":19325,"threads_started":1}
I20260620 14:58:45.159463 9170 tablet_service.cc:1467] Processing CreateTablet for tablet 43d4037423704919a0723a51f7126c15 (DEFAULT_TABLE table=test-table [id=07a81b2b0bd04ff0af216e9bda7234d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:45.160044 9304 tablet_service.cc:1467] Processing CreateTablet for tablet 43d4037423704919a0723a51f7126c15 (DEFAULT_TABLE table=test-table [id=07a81b2b0bd04ff0af216e9bda7234d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:45.161058 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d4037423704919a0723a51f7126c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:45.161054 9170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d4037423704919a0723a51f7126c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:45.160856 9438 tablet_service.cc:1467] Processing CreateTablet for tablet 43d4037423704919a0723a51f7126c15 (DEFAULT_TABLE table=test-table [id=07a81b2b0bd04ff0af216e9bda7234d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:45.162189 9438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43d4037423704919a0723a51f7126c15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:45.172561 9522 tablet_bootstrap.cc:492] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Bootstrap starting.
I20260620 14:58:45.175935 9523 tablet_bootstrap.cc:492] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Bootstrap starting.
I20260620 14:58:45.176945 9522 tablet_bootstrap.cc:654] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:45.178381 9522 log.cc:824] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:45.179641 9523 tablet_bootstrap.cc:654] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:45.181239 9523 log.cc:824] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:45.182230 9524 tablet_bootstrap.cc:492] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Bootstrap starting.
I20260620 14:58:45.185578 9523 tablet_bootstrap.cc:492] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: No bootstrap required, opened a new log
I20260620 14:58:45.185838 9524 tablet_bootstrap.cc:654] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:45.185909 9523 ts_tablet_manager.cc:1397] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 14:58:45.187088 9524 log.cc:824] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:45.188122 9522 tablet_bootstrap.cc:492] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: No bootstrap required, opened a new log
I20260620 14:58:45.188441 9522 ts_tablet_manager.cc:1397] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260620 14:58:45.190284 9524 tablet_bootstrap.cc:492] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: No bootstrap required, opened a new log
I20260620 14:58:45.190618 9524 ts_tablet_manager.cc:1397] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260620 14:58:45.194361 9523 raft_consensus.cc:348] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.194777 9523 raft_consensus.cc:374] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:45.194876 9523 raft_consensus.cc:729] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5525791d26be4d37aa665cd03cc7cd42, State: Initialized, Role: FOLLOWER
I20260620 14:58:45.195614 9523 consensus_queue.cc:260] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.197587 9523 ts_tablet_manager.cc:1428] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 14:58:45.199716 9522 raft_consensus.cc:348] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.200219 9522 raft_consensus.cc:374] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:45.200332 9522 raft_consensus.cc:729] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c675a1ffb5b46af8f1b7e9d6887b06a, State: Initialized, Role: FOLLOWER
I20260620 14:58:45.201009 9522 consensus_queue.cc:260] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.201825 9524 raft_consensus.cc:348] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.202209 9369 heartbeater.cc:503] Master 127.21.159.254:44263 was elected leader, sending a full tablet report...
I20260620 14:58:45.202262 9524 raft_consensus.cc:374] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:45.202356 9524 raft_consensus.cc:729] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370291040d2449db9565068a5b467f26, State: Initialized, Role: FOLLOWER
I20260620 14:58:45.203533 9522 ts_tablet_manager.cc:1428] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260620 14:58:45.205207 9524 consensus_queue.cc:260] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.205660 9530 raft_consensus.cc:482] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:45.206032 9530 raft_consensus.cc:504] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.206519 9503 heartbeater.cc:503] Master 127.21.159.254:44263 was elected leader, sending a full tablet report...
I20260620 14:58:45.208544 9524 ts_tablet_manager.cc:1428] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260620 14:58:45.209625 9530 leader_election.cc:288] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c675a1ffb5b46af8f1b7e9d6887b06a (127.21.159.194:44163), 5525791d26be4d37aa665cd03cc7cd42 (127.21.159.193:37033)
I20260620 14:58:45.215799 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.208670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:44163 (local address 127.21.159.195:42873)
0620 14:58:45.208863 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:45.208883 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:45.209013 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:45.210466 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:45.210472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:45.210500 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:45.210695 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.210705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.212041 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.212044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.213825 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.213839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.215175 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.215179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.215310 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.215334 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:58:45.215444 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:58:45.215559 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:58:45.216641 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.209969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:42873 (local address 127.21.159.194:44163)
0620 14:58:45.210189 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.210195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.210211 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:58:45.210273 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.210278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.210346 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.210415 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:45.210906 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.211904 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.214001 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.214965 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.215187 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.215258 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.216236 (+ 978us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.216377 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:58:45.216650 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.208710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:53051 (local address 127.21.159.193:37033)
0620 14:58:45.208909 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.208915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.209679 (+ 764us) server_negotiation.cc:407] Connection header received
0620 14:58:45.209929 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.209935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.210044 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.210141 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:45.211689 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.212493 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.214608 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.215771 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.215921 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.216010 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.216321 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.216420 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:58:45.217202 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.209086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:37033 (local address 127.21.159.195:53051)
0620 14:58:45.209573 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:45.209587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:45.209730 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:45.211057 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:45.211064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:45.211117 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:45.211533 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.211543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.212639 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.212644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.214445 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:45.214456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.215967 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.215972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:45.216162 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.216201 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:58:45.216289 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:58:45.216405 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":268,"threads_started":1}
I20260620 14:58:45.217607 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d4037423704919a0723a51f7126c15" candidate_uuid: "370291040d2449db9565068a5b467f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" is_pre_election: true
I20260620 14:58:45.218149 9324 raft_consensus.cc:2454] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 370291040d2449db9565068a5b467f26 in term 0.
I20260620 14:58:45.218299 9190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d4037423704919a0723a51f7126c15" candidate_uuid: "370291040d2449db9565068a5b467f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5525791d26be4d37aa665cd03cc7cd42" is_pre_election: true
I20260620 14:58:45.218827 9190 raft_consensus.cc:2454] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 370291040d2449db9565068a5b467f26 in term 0.
I20260620 14:58:45.218830 9392 leader_election.cc:302] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370291040d2449db9565068a5b467f26, 6c675a1ffb5b46af8f1b7e9d6887b06a; no voters:
I20260620 14:58:45.219270 9530 raft_consensus.cc:2781] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:45.219426 9530 raft_consensus.cc:482] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:45.219769 9530 raft_consensus.cc:3037] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:45.222311 9530 raft_consensus.cc:504] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.223258 9530 leader_election.cc:288] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [CANDIDATE]: Term 1 election: Requested vote from peers 6c675a1ffb5b46af8f1b7e9d6887b06a (127.21.159.194:44163), 5525791d26be4d37aa665cd03cc7cd42 (127.21.159.193:37033)
I20260620 14:58:45.223845 9324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d4037423704919a0723a51f7126c15" candidate_uuid: "370291040d2449db9565068a5b467f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a"
I20260620 14:58:45.224082 9324 raft_consensus.cc:3037] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:45.224089 9190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43d4037423704919a0723a51f7126c15" candidate_uuid: "370291040d2449db9565068a5b467f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5525791d26be4d37aa665cd03cc7cd42"
I20260620 14:58:45.224350 9190 raft_consensus.cc:3037] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:58:45.224690 9236 tablet.cc:2369] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:45.226846 9324 raft_consensus.cc:2454] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370291040d2449db9565068a5b467f26 in term 1.
I20260620 14:58:45.227150 9190 raft_consensus.cc:2454] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370291040d2449db9565068a5b467f26 in term 1.
I20260620 14:58:45.227329 9392 leader_election.cc:302] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370291040d2449db9565068a5b467f26, 6c675a1ffb5b46af8f1b7e9d6887b06a; no voters:
I20260620 14:58:45.227653 9530 raft_consensus.cc:2781] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:45.227862 9530 raft_consensus.cc:686] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 1 LEADER]: Becoming Leader. State: Replica: 370291040d2449db9565068a5b467f26, State: Running, Role: LEADER
I20260620 14:58:45.228356 9530 consensus_queue.cc:237] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } }
I20260620 14:58:45.233649 9045 catalog_manager.cc:5495] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 370291040d2449db9565068a5b467f26 (127.21.159.195). New cstate: current_term: 1 leader_uuid: "370291040d2449db9565068a5b467f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "370291040d2449db9565068a5b467f26" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 } health_report { overall_health: UNKNOWN } } }
W20260620 14:58:45.247483 9370 tablet.cc:2369] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:45.251458 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.246112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.21.159.195:46661)
0620 14:58:45.246371 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.246375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.246394 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:58:45.246589 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.246593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.246671 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.246798 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:45.247269 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.248130 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.249436 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.249814 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.249906 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.250349 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:45.250860 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:45.250980 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.251040 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.251166 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 14:58:45.259524 9190 raft_consensus.cc:1261] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 1 FOLLOWER]: Refusing update from remote peer 370291040d2449db9565068a5b467f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:45.259531 9324 raft_consensus.cc:1261] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 1 FOLLOWER]: Refusing update from remote peer 370291040d2449db9565068a5b467f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:45.260509 9531 consensus_queue.cc:1035] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c675a1ffb5b46af8f1b7e9d6887b06a" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:45.260911 9530 consensus_queue.cc:1035] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5525791d26be4d37aa665cd03cc7cd42" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:45.278278 9537 mvcc.cc:205] Tried to move back new op lower bound from 7298938983452925952 to 7298938983340113920. Current Snapshot: MvccSnapshot[applied={T|T < 7298938983452925952}]
I20260620 14:58:45.296084 9045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:45.296448 9045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:45.300597 9045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ce850ae7db24dadaeb4cf35223c2f6c: Sending DeleteTablet for 3 replicas of tablet 43d4037423704919a0723a51f7126c15
I20260620 14:58:45.301829 9438 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d4037423704919a0723a51f7126c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:45 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:60042
I20260620 14:58:45.301959 9170 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d4037423704919a0723a51f7126c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:45 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:37826
I20260620 14:58:45.302657 9304 tablet_service.cc:1514] Processing DeleteTablet for tablet 43d4037423704919a0723a51f7126c15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:45 UTC) from {username='kudu', principal='kudu/127.21.159.254@KRBTEST.COM'} at 127.0.0.1:57296
I20260620 14:58:45.302922 9545 tablet_replica.cc:330] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: stopping tablet replica
I20260620 14:58:45.303505 9545 raft_consensus.cc:2229] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:45.303949 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9107
I20260620 14:58:45.304534 9545 raft_consensus.cc:2258] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:45.312242 9545 ts_tablet_manager.cc:1905] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:45.315402 9547 tablet_replica.cc:330] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: stopping tablet replica
I20260620 14:58:45.316068 9547 raft_consensus.cc:2229] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:45.316551 9546 tablet_replica.cc:330] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: stopping tablet replica
I20260620 14:58:45.316622 9547 raft_consensus.cc:2258] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:45.316959 9546 raft_consensus.cc:2229] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:45.317422 9546 raft_consensus.cc:2258] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 14:58:45 dist-test-slave-3mqx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.193@KRBTEST.COM
I20260620 14:58:45.319875 9546 ts_tablet_manager.cc:1905] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:45.321281 9547 ts_tablet_manager.cc:1905] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:45.322897 9545 ts_tablet_manager.cc:1918] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:45.323118 9545 log.cc:1196] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/43d4037423704919a0723a51f7126c15
I20260620 14:58:45.323714 9545 ts_tablet_manager.cc:1939] T 43d4037423704919a0723a51f7126c15 P 370291040d2449db9565068a5b467f26: Deleting consensus metadata
I20260620 14:58:45.325724 9033 catalog_manager.cc:4841] TS 370291040d2449db9565068a5b467f26 (127.21.159.195:46661): tablet 43d4037423704919a0723a51f7126c15 (table test-table [id=07a81b2b0bd04ff0af216e9bda7234d5]) successfully deleted
I20260620 14:58:45.327296 9546 ts_tablet_manager.cc:1918] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:45.327471 9546 log.cc:1196] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/43d4037423704919a0723a51f7126c15
I20260620 14:58:45.328029 9546 ts_tablet_manager.cc:1939] T 43d4037423704919a0723a51f7126c15 P 5525791d26be4d37aa665cd03cc7cd42: Deleting consensus metadata
I20260620 14:58:45.329941 9030 catalog_manager.cc:4841] TS 5525791d26be4d37aa665cd03cc7cd42 (127.21.159.193:37033): tablet 43d4037423704919a0723a51f7126c15 (table test-table [id=07a81b2b0bd04ff0af216e9bda7234d5]) successfully deleted
I20260620 14:58:45.333060 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.305089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.21.159.193:37033)
0620 14:58:45.305292 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.305301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.305642 (+ 341us) server_negotiation.cc:407] Connection header received
0620 14:58:45.305969 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.305979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.306085 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.306174 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:45.308480 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.310104 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.312903 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.313905 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.314038 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.324668 (+ 10630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:45.324684 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:45.324701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:45.324772 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:45.328719 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.330205 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.330215 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.330219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.330422 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.330945 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.330951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.330954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.331467 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:45.331802 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.332404 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.332666 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:58:45.334463 9230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:45.337350 9547 ts_tablet_manager.cc:1918] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:45.337579 9547 log.cc:1196] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/43d4037423704919a0723a51f7126c15
I20260620 14:58:45.338274 9547 ts_tablet_manager.cc:1939] T 43d4037423704919a0723a51f7126c15 P 6c675a1ffb5b46af8f1b7e9d6887b06a: Deleting consensus metadata
I20260620 14:58:45.340802 9031 catalog_manager.cc:4841] TS 6c675a1ffb5b46af8f1b7e9d6887b06a (127.21.159.194:44163): tablet 43d4037423704919a0723a51f7126c15 (table test-table [id=07a81b2b0bd04ff0af216e9bda7234d5]) successfully deleted
I20260620 14:58:45.433964 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9107
I20260620 14:58:45.452142 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9241
Jun 20 14:58:45 dist-test-slave-3mqx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.194@KRBTEST.COM
I20260620 14:58:45.469774 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.452964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.21.159.194:44163)
0620 14:58:45.453186 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.453192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.453214 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:58:45.453341 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.453347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.453441 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.453562 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:45.454853 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.456068 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.457427 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.458064 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.458197 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.462458 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:45.462478 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:45.462509 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:45.462566 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:45.466505 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.467441 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.467451 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.467455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.467632 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.468018 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.468024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.468028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.468525 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:45.468828 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.469154 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.469383 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:58:45.470530 9364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:45.560508 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9241
I20260620 14:58:45.578488 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9375
Jun 20 14:58:45 dist-test-slave-3mqx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.195@KRBTEST.COM
I20260620 14:58:45.598502 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:45.579454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.21.159.195:46661)
0620 14:58:45.579653 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:45.579660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:45.579684 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:58:45.579790 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:45.579796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:45.579880 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:45.579998 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:45.581569 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.582831 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.584904 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:45.585653 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:45.585817 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:45.590495 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:45.590517 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:45.590538 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:45.590604 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:45.595043 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.596101 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.596118 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.596123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.596352 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:45.596769 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:45.596774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:45.596778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:45.597275 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:45.597655 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:45.597844 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 14:58:45.598114 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:58:45.599433 9498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:45.723882 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9375
I20260620 14:58:45.743575 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9014
I20260620 14:58:45.745054 9076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:45.860875 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9014
2026-06-20T14:58:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4493 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:45Z Disabled control of system clock
I20260620 14:58:45.940192 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:32771
--webserver_interface=127.21.159.254
--webserver_port=0
--builtin_ntp_servers=127.21.159.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:32771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:58:46.287575 9556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:46.288128 9556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:46.288316 9556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:46.296270 9556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:46.296392 9556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:46.296455 9556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:46.296538 9556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:46.296597 9556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:46.306475 9556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:32771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:32771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:46.308080 9556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:46.310045 9556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:46.323208 9561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:46.323289 9562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:46.323249 9564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:46.324194 9556 server_base.cc:1034] running on GCE node
I20260620 14:58:46.325309 9556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:46.326994 9556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:46.328286 9556 hybrid_clock.cc:648] HybridClock initialized: now 1781967526328241 us; error 41 us; skew 500 ppm
I20260620 14:58:46.328677 9556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:46.331321 9556 webserver.cc:460] Webserver started at http://127.21.159.254:32901/ using document root <none> and password file <none>
I20260620 14:58:46.332099 9556 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:46.332243 9556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:46.332588 9556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:46.335884 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "0c7f6da799c1457a96ee94a12fc049e1"
format_stamp: "Formatted at 2026-06-20 14:58:46 on dist-test-slave-3mqx"
server_key: "c353c7646cb2fd09d8887908ce867a30"
server_key_iv: "6710c34b4745a078a0a27008bfc4d2c6"
server_key_version: "encryptionkey@0"
I20260620 14:58:46.336766 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "0c7f6da799c1457a96ee94a12fc049e1"
format_stamp: "Formatted at 2026-06-20 14:58:46 on dist-test-slave-3mqx"
server_key: "c353c7646cb2fd09d8887908ce867a30"
server_key_iv: "6710c34b4745a078a0a27008bfc4d2c6"
server_key_version: "encryptionkey@0"
I20260620 14:58:46.342597 9556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:58:46.346796 9570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:46.348357 9556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:46.348601 9556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "0c7f6da799c1457a96ee94a12fc049e1"
format_stamp: "Formatted at 2026-06-20 14:58:46 on dist-test-slave-3mqx"
server_key: "c353c7646cb2fd09d8887908ce867a30"
server_key_iv: "6710c34b4745a078a0a27008bfc4d2c6"
server_key_version: "encryptionkey@0"
I20260620 14:58:46.348858 9556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:46.365993 9556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:46.366901 9556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:46.367380 9556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:46.387028 9556 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:32771
I20260620 14:58:46.387068 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:32771 every 8 connection(s)
I20260620 14:58:46.388551 9556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:46.392980 9622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:46.396314 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9556
I20260620 14:58:46.396523 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:46.397089 22143 external_mini_cluster.cc:1428] Setting key e979ed4e4698d723f2a25322e4ac501a
I20260620 14:58:46.406637 9622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1: Bootstrap starting.
I20260620 14:58:46.410197 9622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:46.411407 9622 log.cc:824] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:46.411586 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:46.399438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.21.159.254:32771)
0620 14:58:46.400544 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:46.400556 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:46.400602 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:58:46.402307 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:46.402331 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:46.402804 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:46.403276 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:46.403908 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:46.405289 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:46.406690 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:46.407522 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:46.407733 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:46.409054 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:46.409062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:46.409076 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:46.409624 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:46.409716 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:46.409738 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:46.409868 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:46.410086 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:58:46.410477 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":377,"threads_started":1}
I20260620 14:58:46.414367 9622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1: No bootstrap required, opened a new log
I20260620 14:58:46.422163 9622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f6da799c1457a96ee94a12fc049e1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32771 } }
I20260620 14:58:46.422569 9622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:46.422694 9622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7f6da799c1457a96ee94a12fc049e1, State: Initialized, Role: FOLLOWER
I20260620 14:58:46.423362 9622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f6da799c1457a96ee94a12fc049e1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32771 } }
I20260620 14:58:46.423614 9622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:46.423766 9622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:46.423931 9622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:46.426347 9622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f6da799c1457a96ee94a12fc049e1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32771 } }
I20260620 14:58:46.426890 9622 leader_election.cc:302] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c7f6da799c1457a96ee94a12fc049e1; no voters:
I20260620 14:58:46.427520 9622 leader_election.cc:288] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:46.427711 9627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:46.428560 9627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7f6da799c1457a96ee94a12fc049e1, State: Running, Role: LEADER
I20260620 14:58:46.429103 9627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f6da799c1457a96ee94a12fc049e1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32771 } }
I20260620 14:58:46.429693 9622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:46.432204 9628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c7f6da799c1457a96ee94a12fc049e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f6da799c1457a96ee94a12fc049e1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32771 } } }
I20260620 14:58:46.432704 9628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:46.432209 9629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c7f6da799c1457a96ee94a12fc049e1. Latest consensus state: current_term: 1 leader_uuid: "0c7f6da799c1457a96ee94a12fc049e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c7f6da799c1457a96ee94a12fc049e1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 32771 } } }
I20260620 14:58:46.434895 9629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:46.435602 9633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:46.441725 9633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:46.451916 9633 catalog_manager.cc:1347] Generated new cluster ID: bc84f4eca41c434998d3cb11cc7c020e
I20260620 14:58:46.452087 9633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:46.470670 9633 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:46.472491 9633 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:46.481773 9633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1: Generated new TSK 0
I20260620 14:58:46.482661 9633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:46.488160 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.193:0
--local_ip_for_outbound_sockets=127.21.159.193
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32771
--builtin_ntp_servers=127.21.159.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:58:46.850919 9646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:46.851344 9646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:46.851516 9646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:46.859545 9646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:46.859707 9646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:46.859894 9646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.193
I20260620 14:58:46.869556 9646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.159.193
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.193
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:46.871179 9646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:46.873127 9646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:46.887552 9651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:46.887507 9654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:46.887603 9652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:46.892917 9646 server_base.cc:1034] running on GCE node
I20260620 14:58:46.893720 9646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:46.894942 9646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:46.896222 9646 hybrid_clock.cc:648] HybridClock initialized: now 1781967526896149 us; error 76 us; skew 500 ppm
I20260620 14:58:46.896615 9646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:46.899497 9646 webserver.cc:460] Webserver started at http://127.21.159.193:38985/ using document root <none> and password file <none>
I20260620 14:58:46.900270 9646 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:46.900410 9646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:46.900774 9646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:46.903950 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data/instance:
uuid: "e4c30c4096604391a3bee0c4c9745370"
format_stamp: "Formatted at 2026-06-20 14:58:46 on dist-test-slave-3mqx"
server_key: "1dce34fbe105715e79b5477f4aadede6"
server_key_iv: "78d4cf601fc245788037b14050161a0d"
server_key_version: "encryptionkey@0"
I20260620 14:58:46.904875 9646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance:
uuid: "e4c30c4096604391a3bee0c4c9745370"
format_stamp: "Formatted at 2026-06-20 14:58:46 on dist-test-slave-3mqx"
server_key: "1dce34fbe105715e79b5477f4aadede6"
server_key_iv: "78d4cf601fc245788037b14050161a0d"
server_key_version: "encryptionkey@0"
I20260620 14:58:46.910663 9646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:46.915170 9660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:46.916706 9646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:46.916909 9646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal
uuid: "e4c30c4096604391a3bee0c4c9745370"
format_stamp: "Formatted at 2026-06-20 14:58:46 on dist-test-slave-3mqx"
server_key: "1dce34fbe105715e79b5477f4aadede6"
server_key_iv: "78d4cf601fc245788037b14050161a0d"
server_key_version: "encryptionkey@0"
I20260620 14:58:46.917218 9646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:46.938030 9646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:46.939043 9646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:46.939495 9646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:46.940696 9646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:46.942504 9646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:46.942610 9646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:46.942787 9646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:46.942875 9646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:46.979387 9646 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.193:32981
I20260620 14:58:46.979535 9772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.193:32981 every 8 connection(s)
I20260620 14:58:46.980815 9646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/data/info.pb
I20260620 14:58:46.989678 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9646
I20260620 14:58:46.989910 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal/instance
I20260620 14:58:46.990371 22143 external_mini_cluster.cc:1428] Setting key 37e41ed1cb2f5b74539f6d556087c7cc
I20260620 14:58:46.993667 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.194:0
--local_ip_for_outbound_sockets=127.21.159.194
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32771
--builtin_ntp_servers=127.21.159.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:58:46.996181 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:46.983873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:56355 (local address 127.21.159.254:32771)
0620 14:58:46.984490 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:46.984495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:46.985911 (+ 1416us) server_negotiation.cc:407] Connection header received
0620 14:58:46.987936 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:46.987942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:46.988058 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:46.988190 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:46.989466 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:46.990298 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:46.992299 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:46.993046 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:46.993185 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:46.995046 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:46.995077 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:46.995084 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:46.995114 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:46.995171 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:46.995192 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:46.995294 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:46.995576 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:58:46.995708 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":378,"threads_started":1}
I20260620 14:58:46.997295 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:46.984682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32771 (local address 127.21.159.193:56355)
0620 14:58:46.985668 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:46.985708 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:46.987608 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:46.988492 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:46.988502 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:46.988609 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:46.989276 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:46.989295 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:46.990493 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:46.990499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:46.992119 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:46.992140 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:46.993312 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:46.993318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:46.993571 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:46.994559 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:46.994562 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:46.994617 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:46.994738 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:46.995323 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:46.995334 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:46.995342 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:58:46.995528 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 14:58:46.995970 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":288,"threads_started":1}
I20260620 14:58:46.999202 9773 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32771
I20260620 14:58:46.999606 9773 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:47.000681 9773 heartbeater.cc:511] Master 127.21.159.254:32771 requested a full tablet report, sending...
I20260620 14:58:47.003609 9587 ts_manager.cc:194] Registered new tserver with Master: e4c30c4096604391a3bee0c4c9745370 (127.21.159.193:32981)
I20260620 14:58:47.006009 9587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.193:56355
W20260620 14:58:47.337759 9777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:47.338078 9777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:47.338167 9777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:47.345888 9777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:47.346017 9777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:47.346130 9777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.194
I20260620 14:58:47.355965 9777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.159.194
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.194
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:47.357298 9777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:47.359124 9777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:47.373064 9783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:47.373090 9785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:47.373730 9782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:47.374554 9777 server_base.cc:1034] running on GCE node
I20260620 14:58:47.375324 9777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:47.376390 9777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:47.377626 9777 hybrid_clock.cc:648] HybridClock initialized: now 1781967527377579 us; error 42 us; skew 500 ppm
I20260620 14:58:47.377945 9777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:47.380592 9777 webserver.cc:460] Webserver started at http://127.21.159.194:34525/ using document root <none> and password file <none>
I20260620 14:58:47.381317 9777 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:47.381412 9777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:47.381690 9777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:47.384842 9777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data/instance:
uuid: "074253d6cb414d78870e05c8c5554f68"
format_stamp: "Formatted at 2026-06-20 14:58:47 on dist-test-slave-3mqx"
server_key: "b895e2397413fb1b6fe73e67c4b688dc"
server_key_iv: "df0e954b3f014f47a8edb9915ad6d475"
server_key_version: "encryptionkey@0"
I20260620 14:58:47.385696 9777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance:
uuid: "074253d6cb414d78870e05c8c5554f68"
format_stamp: "Formatted at 2026-06-20 14:58:47 on dist-test-slave-3mqx"
server_key: "b895e2397413fb1b6fe73e67c4b688dc"
server_key_iv: "df0e954b3f014f47a8edb9915ad6d475"
server_key_version: "encryptionkey@0"
I20260620 14:58:47.391590 9777 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 14:58:47.395604 9791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:47.397229 9777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:58:47.397447 9777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal
uuid: "074253d6cb414d78870e05c8c5554f68"
format_stamp: "Formatted at 2026-06-20 14:58:47 on dist-test-slave-3mqx"
server_key: "b895e2397413fb1b6fe73e67c4b688dc"
server_key_iv: "df0e954b3f014f47a8edb9915ad6d475"
server_key_version: "encryptionkey@0"
I20260620 14:58:47.397696 9777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:47.420452 9777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:47.421415 9777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:47.421741 9777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:47.422850 9777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:47.424420 9777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:47.424554 9777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:47.424691 9777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:47.424778 9777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:47.459805 9777 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.194:45783
I20260620 14:58:47.459900 9903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.194:45783 every 8 connection(s)
I20260620 14:58:47.461218 9777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/data/info.pb
I20260620 14:58:47.463492 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9777
I20260620 14:58:47.463673 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal/instance
I20260620 14:58:47.464233 22143 external_mini_cluster.cc:1428] Setting key 92bfc8135e39d13145cd144dee9ca2f6
I20260620 14:58:47.467749 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.159.195:0
--local_ip_for_outbound_sockets=127.21.159.195
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32771
--builtin_ntp_servers=127.21.159.212:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:58:47.477111 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:47.464116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.194:40969 (local address 127.21.159.254:32771)
0620 14:58:47.464342 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:47.464348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:47.466088 (+ 1740us) server_negotiation.cc:407] Connection header received
0620 14:58:47.468733 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:47.468737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:47.468807 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:47.468895 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:47.470400 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.471208 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.473749 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.474179 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.474281 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:47.476033 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:47.476039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:47.476042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:47.476062 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:47.476094 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:47.476106 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:47.476195 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:47.476472 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:58:47.476634 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:58:47.478056 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:47.464740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32771 (local address 127.21.159.194:40969)
0620 14:58:47.465900 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:47.465950 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:47.468390 (+ 2440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:47.469196 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:47.469211 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:47.469352 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:47.470217 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:47.470245 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.471366 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.471374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:47.473574 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:47.473593 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.474420 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.474430 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:47.474788 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:47.475608 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:47.475614 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:47.475681 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:47.475799 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:47.476217 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:47.476231 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:47.476242 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:58:47.476459 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 14:58:47.476842 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":325,"threads_started":1}
I20260620 14:58:47.479638 9904 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32771
I20260620 14:58:47.479980 9904 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:47.480770 9904 heartbeater.cc:511] Master 127.21.159.254:32771 requested a full tablet report, sending...
I20260620 14:58:47.482420 9587 ts_manager.cc:194] Registered new tserver with Master: 074253d6cb414d78870e05c8c5554f68 (127.21.159.194:45783)
I20260620 14:58:47.483739 9587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.194:40969
W20260620 14:58:47.827481 9907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:47.827787 9907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:47.827891 9907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:47.835963 9907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:47.836090 9907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:47.836223 9907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.159.195
I20260620 14:58:47.845933 9907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.159.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.159.195
--webserver_port=0
--tserver_master_addrs=127.21.159.254:32771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.159.195
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:47.847380 9907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:47.849232 9907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:47.863628 9913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:47.863575 9915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:47.863552 9912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:47.864363 9907 server_base.cc:1034] running on GCE node
I20260620 14:58:47.865020 9907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:47.866072 9907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:47.867285 9907 hybrid_clock.cc:648] HybridClock initialized: now 1781967527867240 us; error 30 us; skew 500 ppm
I20260620 14:58:47.867605 9907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:58:47.870177 9907 webserver.cc:460] Webserver started at http://127.21.159.195:46295/ using document root <none> and password file <none>
I20260620 14:58:47.870894 9907 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:47.870985 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:47.871316 9907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:47.874320 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data/instance:
uuid: "e2f379b456e54d67af10da899bbca147"
format_stamp: "Formatted at 2026-06-20 14:58:47 on dist-test-slave-3mqx"
server_key: "95ee9df2cec691c42053186f54a347d4"
server_key_iv: "b38e3985f90548ca5fe9cf31060dd7be"
server_key_version: "encryptionkey@0"
I20260620 14:58:47.875200 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance:
uuid: "e2f379b456e54d67af10da899bbca147"
format_stamp: "Formatted at 2026-06-20 14:58:47 on dist-test-slave-3mqx"
server_key: "95ee9df2cec691c42053186f54a347d4"
server_key_iv: "b38e3985f90548ca5fe9cf31060dd7be"
server_key_version: "encryptionkey@0"
I20260620 14:58:47.880630 9907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:58:47.884562 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:47.886355 9907 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:58:47.886559 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal
uuid: "e2f379b456e54d67af10da899bbca147"
format_stamp: "Formatted at 2026-06-20 14:58:47 on dist-test-slave-3mqx"
server_key: "95ee9df2cec691c42053186f54a347d4"
server_key_iv: "b38e3985f90548ca5fe9cf31060dd7be"
server_key_version: "encryptionkey@0"
I20260620 14:58:47.886832 9907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:47.916844 9907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:47.917874 9907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:47.918210 9907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:47.919411 9907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:58:47.921079 9907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:58:47.921196 9907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:47.921298 9907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:58:47.921356 9907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:47.956374 9907 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.195:33829
I20260620 14:58:47.956475 10033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.195:33829 every 8 connection(s)
I20260620 14:58:47.957744 9907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/data/info.pb
I20260620 14:58:47.958570 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 9907
I20260620 14:58:47.958773 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal/instance
I20260620 14:58:47.959348 22143 external_mini_cluster.cc:1428] Setting key bfc4b7d8e4ecbbee0a7932457e896dfe
I20260620 14:58:47.971570 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:47.960663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:51881 (local address 127.21.159.254:32771)
0620 14:58:47.960883 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:47.960889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:47.962778 (+ 1889us) server_negotiation.cc:407] Connection header received
0620 14:58:47.964707 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:47.964711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:47.964782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:47.964882 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:47.965998 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.966733 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.968621 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.969103 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.969205 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:47.970648 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:47.970655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:47.970658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:47.970679 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:47.970715 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:47.970727 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:47.970815 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:47.971186 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:58:47.971281 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:58:47.972630 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:47.961453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:32771 (local address 127.21.159.195:51881)
0620 14:58:47.962621 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:47.962667 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:47.964439 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:47.965132 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:47.965140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:47.965246 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:58:47.965832 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:47.965849 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.966863 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.966904 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:47.968482 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:47.968496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.969310 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.969315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:47.969539 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:47.970279 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:58:47.970281 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:58:47.970324 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:58:47.970419 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:58:47.970826 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:58:47.970834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:58:47.970840 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:58:47.971043 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:58:47.971487 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":364,"threads_started":1}
I20260620 14:58:47.974262 10034 heartbeater.cc:347] Connected to a master server at 127.21.159.254:32771
I20260620 14:58:47.974598 10034 heartbeater.cc:464] Registering TS with master...
I20260620 14:58:47.975469 10034 heartbeater.cc:511] Master 127.21.159.254:32771 requested a full tablet report, sending...
I20260620 14:58:47.976972 9586 ts_manager.cc:194] Registered new tserver with Master: e2f379b456e54d67af10da899bbca147 (127.21.159.195:33829)
I20260620 14:58:47.977843 9586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.159.195:51881
I20260620 14:58:47.987848 22143 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:58:47.998863 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:47.992360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.21.159.254:32771)
0620 14:58:47.992685 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:47.992692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:47.993083 (+ 391us) server_negotiation.cc:407] Connection header received
0620 14:58:47.993326 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:47.993330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:47.993400 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:47.993523 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:47.994060 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.994806 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.996315 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:47.996747 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:47.996892 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:47.998072 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:47.998078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:47.998081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:47.998102 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:47.998139 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:47.998152 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:47.998256 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:47.998434 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:58:47.998555 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:58:48.003649 9586 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:58:48.009893 9773 heartbeater.cc:503] Master 127.21.159.254:32771 was elected leader, sending a full tablet report...
W20260620 14:58:48.011543 9586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:58:48.051258 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.026240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:32981 (local address 127.0.0.1:53634)
0620 14:58:48.042228 (+ 15988us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:48.042251 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:48.042472 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:48.043258 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:48.043265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:48.043306 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:48.043736 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.043745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.045455 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.045462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.047909 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.047928 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.050160 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.050166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.050391 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.050446 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:58:48.050586 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 14:58:48.050813 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15838,"mutex_wait_us":161,"thread_start_us":15696,"threads_started":1}
I20260620 14:58:48.051365 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.027010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.21.159.193:32981)
0620 14:58:48.033692 (+ 6682us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.033708 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.042364 (+ 8656us) server_negotiation.cc:407] Connection header received
0620 14:58:48.042671 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.042683 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.042918 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.043112 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:48.043920 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.045319 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.048148 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.049962 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.050204 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.050462 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.050690 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.050903 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6411,"thread_start_us":6174,"threads_started":1}
I20260620 14:58:48.051934 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.025687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.21.159.195:33829)
0620 14:58:48.042001 (+ 16314us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.042014 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.042053 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:58:48.042153 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.042164 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.042417 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.042619 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:48.043598 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.045156 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.048567 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.050031 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.050224 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.050461 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.051365 (+ 904us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.051568 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16031,"thread_start_us":15792,"threads_started":1}
I20260620 14:58:48.052018 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.026281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.21.159.194:45783)
0620 14:58:48.039714 (+ 13433us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.039729 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.041531 (+ 1802us) server_negotiation.cc:407] Connection header received
0620 14:58:48.041657 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.041667 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.042025 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.042262 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:48.043416 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.045126 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.048540 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.050078 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.050251 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.050462 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.051422 (+ 960us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.051622 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13149,"thread_start_us":13468,"threads_started":1}
I20260620 14:58:48.052486 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.025365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:33829 (local address 127.0.0.1:33128)
0620 14:58:48.040713 (+ 15348us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:48.040770 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:48.041049 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:48.042734 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:48.042741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:48.042777 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:48.043372 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.043387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.045317 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.045322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.048293 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.048309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.050620 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.050626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.050862 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.050909 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:58:48.050983 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:58:48.051129 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15006,"mutex_wait_us":225,"thread_start_us":15373,"threads_started":1}
I20260620 14:58:48.052696 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.025803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45783 (local address 127.0.0.1:37766)
0620 14:58:48.041098 (+ 15295us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:48.041132 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:48.041341 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:48.042572 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:48.042583 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:48.042614 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:48.043118 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.043138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.045317 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.045322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.048293 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.048309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.050652 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.050659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.050913 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.050949 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:58:48.051036 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:58:48.051225 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15134,"mutex_wait_us":41,"thread_start_us":14388,"threads_started":1}
I20260620 14:58:48.056298 9839 tablet_service.cc:1467] Processing CreateTablet for tablet d44f4b99f93240688bb1e2fa4fcd89e3 (DEFAULT_TABLE table=test-table [id=a39fc1ded1cc4eb6a587113e3e7c2b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:48.056352 9708 tablet_service.cc:1467] Processing CreateTablet for tablet d44f4b99f93240688bb1e2fa4fcd89e3 (DEFAULT_TABLE table=test-table [id=a39fc1ded1cc4eb6a587113e3e7c2b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:48.056701 9969 tablet_service.cc:1467] Processing CreateTablet for tablet d44f4b99f93240688bb1e2fa4fcd89e3 (DEFAULT_TABLE table=test-table [id=a39fc1ded1cc4eb6a587113e3e7c2b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:58:48.057822 9708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f4b99f93240688bb1e2fa4fcd89e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:48.057861 9839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f4b99f93240688bb1e2fa4fcd89e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:48.058553 9969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d44f4b99f93240688bb1e2fa4fcd89e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:48.073346 10052 tablet_bootstrap.cc:492] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Bootstrap starting.
I20260620 14:58:48.073547 10054 tablet_bootstrap.cc:492] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Bootstrap starting.
I20260620 14:58:48.074426 10053 tablet_bootstrap.cc:492] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Bootstrap starting.
I20260620 14:58:48.077754 10054 tablet_bootstrap.cc:654] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:48.077875 10052 tablet_bootstrap.cc:654] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:48.078485 10053 tablet_bootstrap.cc:654] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:48.079097 10054 log.cc:824] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:48.079331 10052 log.cc:824] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:48.079926 10053 log.cc:824] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:48.083460 10052 tablet_bootstrap.cc:492] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: No bootstrap required, opened a new log
I20260620 14:58:48.083451 10053 tablet_bootstrap.cc:492] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: No bootstrap required, opened a new log
I20260620 14:58:48.083443 10054 tablet_bootstrap.cc:492] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: No bootstrap required, opened a new log
I20260620 14:58:48.083730 10054 ts_tablet_manager.cc:1397] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 14:58:48.083741 10052 ts_tablet_manager.cc:1397] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 14:58:48.083861 10053 ts_tablet_manager.cc:1397] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 14:58:48.093659 10053 raft_consensus.cc:348] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.093691 10054 raft_consensus.cc:348] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.094141 10053 raft_consensus.cc:374] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:48.094249 10054 raft_consensus.cc:374] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:48.094266 10053 raft_consensus.cc:729] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074253d6cb414d78870e05c8c5554f68, State: Initialized, Role: FOLLOWER
I20260620 14:58:48.094425 10054 raft_consensus.cc:729] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f379b456e54d67af10da899bbca147, State: Initialized, Role: FOLLOWER
I20260620 14:58:48.095172 10053 consensus_queue.cc:260] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.095172 10054 consensus_queue.cc:260] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.096463 9904 heartbeater.cc:503] Master 127.21.159.254:32771 was elected leader, sending a full tablet report...
I20260620 14:58:48.097048 10053 ts_tablet_manager.cc:1428] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 14:58:48.097136 10052 raft_consensus.cc:348] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.097678 10052 raft_consensus.cc:374] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:48.097792 10052 raft_consensus.cc:729] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c30c4096604391a3bee0c4c9745370, State: Initialized, Role: FOLLOWER
I20260620 14:58:48.098835 10052 consensus_queue.cc:260] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.103520 10034 heartbeater.cc:503] Master 127.21.159.254:32771 was elected leader, sending a full tablet report...
I20260620 14:58:48.104482 10054 ts_tablet_manager.cc:1428] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Time spent starting tablet: real 0.021s user 0.014s sys 0.004s
I20260620 14:58:48.105803 10052 ts_tablet_manager.cc:1428] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Time spent starting tablet: real 0.022s user 0.018s sys 0.000s
I20260620 14:58:48.164458 10060 raft_consensus.cc:482] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:48.164851 10060 raft_consensus.cc:504] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.166774 10060 leader_election.cc:288] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f379b456e54d67af10da899bbca147 (127.21.159.195:33829), 074253d6cb414d78870e05c8c5554f68 (127.21.159.194:45783)
I20260620 14:58:48.172020 10059 raft_consensus.cc:482] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:58:48.172336 10059 raft_consensus.cc:504] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.174293 10059 leader_election.cc:288] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074253d6cb414d78870e05c8c5554f68 (127.21.159.194:45783), e4c30c4096604391a3bee0c4c9745370 (127.21.159.193:32981)
I20260620 14:58:48.175107 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.167250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:45273 (local address 127.21.159.194:45783)
0620 14:58:48.167479 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.167489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.168345 (+ 856us) server_negotiation.cc:407] Connection header received
0620 14:58:48.168591 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.168597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.168679 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.168730 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:48.169470 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.170248 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.172663 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.173957 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.174132 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.174218 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.174650 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.174804 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:58:48.175225 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.167339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.193:39841 (local address 127.21.159.195:33829)
0620 14:58:48.167574 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.167583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.168506 (+ 923us) server_negotiation.cc:407] Connection header received
0620 14:58:48.168714 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.168720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.168801 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.168913 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:48.169625 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.170419 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.172752 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.174095 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.174285 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.174365 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.174813 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.174947 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:58:48.175920 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.167132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45783 (local address 127.21.159.193:45273)
0620 14:58:48.168192 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:48.168216 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:48.168409 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:48.168889 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:48.168892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:48.168917 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:48.169307 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.169316 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.170409 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.170414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.172442 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.172460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.174178 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.174184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.174437 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.174477 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:58:48.175303 (+ 826us) client_negotiation.cc:240] Negotiation successful
0620 14:58:48.175470 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":173,"thread_start_us":596,"threads_started":1}
I20260620 14:58:48.176661 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.167241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.195:33829 (local address 127.21.159.193:39841)
0620 14:58:48.168382 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:48.168400 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:48.168565 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:48.168927 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:48.168932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:48.168954 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:48.169419 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.169432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.170585 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.170590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.172479 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.172494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.174370 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.174378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.174602 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.174641 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:58:48.175378 (+ 737us) client_negotiation.cc:240] Negotiation successful
0620 14:58:48.175529 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":35,"thread_start_us":565,"threads_started":1}
I20260620 14:58:48.177181 9859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f4b99f93240688bb1e2fa4fcd89e3" candidate_uuid: "e4c30c4096604391a3bee0c4c9745370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074253d6cb414d78870e05c8c5554f68" is_pre_election: true
I20260620 14:58:48.177675 9989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f4b99f93240688bb1e2fa4fcd89e3" candidate_uuid: "e4c30c4096604391a3bee0c4c9745370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f379b456e54d67af10da899bbca147" is_pre_election: true
I20260620 14:58:48.177855 9859 raft_consensus.cc:2454] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c30c4096604391a3bee0c4c9745370 in term 0.
I20260620 14:58:48.178099 9989 raft_consensus.cc:2454] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c30c4096604391a3bee0c4c9745370 in term 0.
I20260620 14:58:48.178645 9663 leader_election.cc:302] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074253d6cb414d78870e05c8c5554f68, e4c30c4096604391a3bee0c4c9745370; no voters:
I20260620 14:58:48.179358 10060 raft_consensus.cc:2781] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:58:48.179561 10060 raft_consensus.cc:482] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:58:48.179682 10060 raft_consensus.cc:3037] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:48.182057 10060 raft_consensus.cc:504] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.182833 10060 leader_election.cc:288] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [CANDIDATE]: Term 1 election: Requested vote from peers e2f379b456e54d67af10da899bbca147 (127.21.159.195:33829), 074253d6cb414d78870e05c8c5554f68 (127.21.159.194:45783)
I20260620 14:58:48.182998 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.174865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.194:45783 (local address 127.21.159.195:38037)
0620 14:58:48.175311 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:48.175330 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:48.175440 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:48.176187 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:48.176192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:48.176230 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:48.176515 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.176530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.178612 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.178620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.180462 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.180483 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.182177 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.182183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.182381 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.182421 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:58:48.182497 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:58:48.182665 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315}
I20260620 14:58:48.183269 9989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f4b99f93240688bb1e2fa4fcd89e3" candidate_uuid: "e4c30c4096604391a3bee0c4c9745370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f379b456e54d67af10da899bbca147"
I20260620 14:58:48.184000 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.175337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:38037 (local address 127.21.159.194:45783)
0620 14:58:48.175844 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.175850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.175870 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:58:48.175929 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.175936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.176040 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.177131 (+ 1091us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:48.177273 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.178452 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.180676 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.181994 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.183424 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.183497 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.183549 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.183685 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260620 14:58:48.183876 9859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f4b99f93240688bb1e2fa4fcd89e3" candidate_uuid: "e4c30c4096604391a3bee0c4c9745370" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074253d6cb414d78870e05c8c5554f68"
I20260620 14:58:48.184342 9859 raft_consensus.cc:3037] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:48.184633 9858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f4b99f93240688bb1e2fa4fcd89e3" candidate_uuid: "e2f379b456e54d67af10da899bbca147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074253d6cb414d78870e05c8c5554f68" is_pre_election: true
I20260620 14:58:48.185096 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.175770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.159.195:41715 (local address 127.21.159.193:32981)
0620 14:58:48.176115 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.176124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.176760 (+ 636us) server_negotiation.cc:407] Connection header received
0620 14:58:48.177022 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.177028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.177125 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.177186 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:58:48.179253 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.180351 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.182775 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.184015 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.184149 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.184229 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.184755 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.184883 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260620 14:58:48.185179 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.174898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.193:32981 (local address 127.21.159.195:41715)
0620 14:58:48.176664 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0620 14:58:48.176684 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:58:48.176849 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:58:48.178479 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:58:48.178483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:58:48.178507 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:58:48.179015 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.179031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.180519 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.180528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.182547 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:58:48.182558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.184386 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.184394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:58:48.184565 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.184599 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:58:48.184729 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:58:48.184863 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1638,"thread_start_us":1404,"threads_started":1}
I20260620 14:58:48.185415 9989 raft_consensus.cc:3037] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:48.186594 9728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d44f4b99f93240688bb1e2fa4fcd89e3" candidate_uuid: "e2f379b456e54d67af10da899bbca147" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c30c4096604391a3bee0c4c9745370" is_pre_election: true
I20260620 14:58:48.187143 9728 raft_consensus.cc:2379] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2f379b456e54d67af10da899bbca147 in current term 1: Already voted for candidate e4c30c4096604391a3bee0c4c9745370 in this term.
I20260620 14:58:48.187822 9859 raft_consensus.cc:2454] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c30c4096604391a3bee0c4c9745370 in term 1.
I20260620 14:58:48.188040 9922 leader_election.cc:302] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f379b456e54d67af10da899bbca147; no voters: 074253d6cb414d78870e05c8c5554f68, e4c30c4096604391a3bee0c4c9745370
I20260620 14:58:48.188218 9663 leader_election.cc:302] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074253d6cb414d78870e05c8c5554f68, e4c30c4096604391a3bee0c4c9745370; no voters:
I20260620 14:58:48.188423 9989 raft_consensus.cc:2454] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c30c4096604391a3bee0c4c9745370 in term 1.
I20260620 14:58:48.188563 10059 raft_consensus.cc:2726] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:58:48.188562 10060 raft_consensus.cc:2781] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:48.189038 10060 raft_consensus.cc:686] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 1 LEADER]: Becoming Leader. State: Replica: e4c30c4096604391a3bee0c4c9745370, State: Running, Role: LEADER
I20260620 14:58:48.189631 10060 consensus_queue.cc:237] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } }
I20260620 14:58:48.194520 9587 catalog_manager.cc:5495] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c30c4096604391a3bee0c4c9745370 (127.21.159.193). New cstate: current_term: 1 leader_uuid: "e4c30c4096604391a3bee0c4c9745370" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c30c4096604391a3bee0c4c9745370" member_type: VOTER last_known_addr { host: "127.21.159.193" port: 32981 } health_report { overall_health: HEALTHY } } }
W20260620 14:58:48.209632 10035 tablet.cc:2369] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:58:48.213395 9905 tablet.cc:2369] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:48.222393 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.216016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.21.159.193:32981)
0620 14:58:48.216255 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.216264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.216363 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:58:48.216629 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.216642 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.216740 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.216917 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:58:48.217293 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.218083 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.219595 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.220048 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.220191 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.220714 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:58:48.221594 (+ 880us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:58:48.221815 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.221928 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.222024 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:58:48.231985 9859 raft_consensus.cc:1261] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 1 FOLLOWER]: Refusing update from remote peer e4c30c4096604391a3bee0c4c9745370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:48.232053 9989 raft_consensus.cc:1261] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 1 FOLLOWER]: Refusing update from remote peer e4c30c4096604391a3bee0c4c9745370: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:58:48.232999 10060 consensus_queue.cc:1035] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074253d6cb414d78870e05c8c5554f68" member_type: VOTER last_known_addr { host: "127.21.159.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:58:48.233306 9774 tablet.cc:2369] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:58:48.233361 10064 consensus_queue.cc:1035] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f379b456e54d67af10da899bbca147" member_type: VOTER last_known_addr { host: "127.21.159.195" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:58:48.246920 10067 mvcc.cc:205] Tried to move back new op lower bound from 7298938995625820160 to 7298938995468832768. Current Snapshot: MvccSnapshot[applied={T|T < 7298938995625820160}]
I20260620 14:58:48.266476 9587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:48.266814 9587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:58:48.271276 9587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c7f6da799c1457a96ee94a12fc049e1: Sending DeleteTablet for 3 replicas of tablet d44f4b99f93240688bb1e2fa4fcd89e3
I20260620 14:58:48.272668 9969 tablet_service.cc:1514] Processing DeleteTablet for tablet d44f4b99f93240688bb1e2fa4fcd89e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:48 UTC) from {username='slave'} at 127.0.0.1:33128
I20260620 14:58:48.272719 9708 tablet_service.cc:1514] Processing DeleteTablet for tablet d44f4b99f93240688bb1e2fa4fcd89e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:48 UTC) from {username='slave'} at 127.0.0.1:53634
I20260620 14:58:48.273625 9839 tablet_service.cc:1514] Processing DeleteTablet for tablet d44f4b99f93240688bb1e2fa4fcd89e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:58:48 UTC) from {username='slave'} at 127.0.0.1:37766
I20260620 14:58:48.273881 10078 tablet_replica.cc:330] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: stopping tablet replica
I20260620 14:58:48.274442 10078 raft_consensus.cc:2229] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:58:48.274705 10079 tablet_replica.cc:330] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: stopping tablet replica
I20260620 14:58:48.275245 10079 raft_consensus.cc:2229] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:48.275250 10078 raft_consensus.cc:2258] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:48.275636 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9646
I20260620 14:58:48.275956 10079 raft_consensus.cc:2258] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:48.276511 10077 tablet_replica.cc:330] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: stopping tablet replica
I20260620 14:58:48.277024 10077 raft_consensus.cc:2229] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:58:48.277601 10077 raft_consensus.cc:2258] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:58:48.278534 10079 ts_tablet_manager.cc:1905] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:48.279541 10078 ts_tablet_manager.cc:1905] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:48.286257 10079 ts_tablet_manager.cc:1918] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:48.286592 10079 log.cc:1196] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-1/wal/wals/d44f4b99f93240688bb1e2fa4fcd89e3
I20260620 14:58:48.287329 10079 ts_tablet_manager.cc:1939] T d44f4b99f93240688bb1e2fa4fcd89e3 P 074253d6cb414d78870e05c8c5554f68: Deleting consensus metadata
I20260620 14:58:48.289119 10077 ts_tablet_manager.cc:1905] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:58:48.289479 9573 catalog_manager.cc:4841] TS 074253d6cb414d78870e05c8c5554f68 (127.21.159.194:45783): tablet d44f4b99f93240688bb1e2fa4fcd89e3 (table test-table [id=a39fc1ded1cc4eb6a587113e3e7c2b2a]) successfully deleted
I20260620 14:58:48.294222 10078 ts_tablet_manager.cc:1918] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:48.294564 10078 log.cc:1196] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-0/wal/wals/d44f4b99f93240688bb1e2fa4fcd89e3
I20260620 14:58:48.295300 10078 ts_tablet_manager.cc:1939] T d44f4b99f93240688bb1e2fa4fcd89e3 P e4c30c4096604391a3bee0c4c9745370: Deleting consensus metadata
I20260620 14:58:48.297552 10077 ts_tablet_manager.cc:1918] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:58:48.297570 9571 catalog_manager.cc:4841] TS e4c30c4096604391a3bee0c4c9745370 (127.21.159.193:32981): tablet d44f4b99f93240688bb1e2fa4fcd89e3 (table test-table [id=a39fc1ded1cc4eb6a587113e3e7c2b2a]) successfully deleted
I20260620 14:58:48.297753 10077 log.cc:1196] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Deleting WAL directory at /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967200138920-22143-0/minicluster-data/ts-2/wal/wals/d44f4b99f93240688bb1e2fa4fcd89e3
I20260620 14:58:48.298341 10051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.279277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.21.159.193:32981)
0620 14:58:48.279506 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.279515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.288039 (+ 8524us) server_negotiation.cc:407] Connection header received
0620 14:58:48.288608 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.288615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.288761 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.288872 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:48.289617 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.290937 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.293374 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.294173 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.294293 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.296066 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:48.296079 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:48.296099 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:48.297007 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:48.297116 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:48.297158 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:48.297262 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.297720 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.297881 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:58:48.298353 10077 ts_tablet_manager.cc:1939] T d44f4b99f93240688bb1e2fa4fcd89e3 P e2f379b456e54d67af10da899bbca147: Deleting consensus metadata
I20260620 14:58:48.299731 9768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:48.300477 9572 catalog_manager.cc:4841] TS e2f379b456e54d67af10da899bbca147 (127.21.159.195:33829): tablet d44f4b99f93240688bb1e2fa4fcd89e3 (table test-table [id=a39fc1ded1cc4eb6a587113e3e7c2b2a]) successfully deleted
I20260620 14:58:48.406297 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9646
I20260620 14:58:48.425673 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9777
I20260620 14:58:48.432961 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.426597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.21.159.194:45783)
0620 14:58:48.426770 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.426774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.426903 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:58:48.427096 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.427108 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.427201 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.427292 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:48.427716 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.428450 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.429918 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.430328 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.430425 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.431665 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:48.431675 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:48.431690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:48.432283 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:48.432333 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:48.432352 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:48.432443 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.432603 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.432689 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:58:48.433665 9899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:48.523809 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9777
I20260620 14:58:48.543540 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9907
I20260620 14:58:48.550981 10048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:48.544391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.21.159.195:33829)
0620 14:58:48.544622 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:48.544629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:48.544652 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:58:48.544893 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:48.544899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:48.545008 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:48.545132 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:48.545559 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.546338 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.547750 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:48.548178 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:48.548315 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:48.549505 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:48.549514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:48.549528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:48.550176 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:48.550225 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:48.550245 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:48.550388 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:48.550546 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:58:48.550658 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 14:58:48.551869 10029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:48.644542 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9907
I20260620 14:58:48.661841 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 9556
I20260620 14:58:48.662899 9617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:48.773191 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 9556
2026-06-20T14:58:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2915 ms)
[----------] 6 tests from EncryptionPolicyTest (22834 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:48 dist-test-slave-3mqx krb5kdc[10087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:48 dist-test-slave-3mqx krb5kdc[10087](info): set up 2 sockets
Jun 20 14:58:48 dist-test-slave-3mqx krb5kdc[10087](info): commencing operation
krb5kdc: starting...
W20260620 14:58:49.955734 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:50 dist-test-slave-3mqx krb5kdc[10087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:58:50.227344 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:50.593262 10100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:50.593580 10100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:50.593676 10100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:50.601341 10100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:50.601495 10100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:50.601543 10100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:50.601599 10100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:50.601647 10100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:50.612285 10100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:50.613758 10100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:50.615146 10100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:50.628469 10108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:50.628551 10105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:50.628489 10106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:50.628849 10100 server_base.cc:1034] running on GCE node
I20260620 14:58:50.629992 10100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:50.631758 10100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:50.633020 10100 hybrid_clock.cc:648] HybridClock initialized: now 1781967530632975 us; error 59 us; skew 500 ppm
I20260620 14:58:50.633409 10100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:50 dist-test-slave-3mqx krb5kdc[10087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:50.636941 10100 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:58:50.638741 10100 webserver.cc:460] Webserver started at http://127.0.0.1:45151/ using document root <none> and password file <none>
I20260620 14:58:50.639628 10100 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:50.639807 10100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:50.640134 10100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:50.643566 10100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f"
format_stamp: "Formatted at 2026-06-20 14:58:50 on dist-test-slave-3mqx"
server_key: "38fa8f4c549b8ef6bffb309bad58c3f0"
server_key_iv: "60ad30c00ae261e7a32ed3a867c0258b"
server_key_version: "encryptionkey@0"
I20260620 14:58:50.644500 10100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f"
format_stamp: "Formatted at 2026-06-20 14:58:50 on dist-test-slave-3mqx"
server_key: "38fa8f4c549b8ef6bffb309bad58c3f0"
server_key_iv: "60ad30c00ae261e7a32ed3a867c0258b"
server_key_version: "encryptionkey@0"
I20260620 14:58:50.650439 10100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:58:50.654775 10115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:50.656591 10100 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:58:50.656862 10100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f"
format_stamp: "Formatted at 2026-06-20 14:58:50 on dist-test-slave-3mqx"
server_key: "38fa8f4c549b8ef6bffb309bad58c3f0"
server_key_iv: "60ad30c00ae261e7a32ed3a867c0258b"
server_key_version: "encryptionkey@0"
I20260620 14:58:50.657116 10100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:50.674944 10100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:50.680123 10100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:50.680464 10100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:50.700785 10100 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45345
I20260620 14:58:50.700814 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45345 every 8 connection(s)
I20260620 14:58:50.702356 10100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:50.705351 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10100
I20260620 14:58:50.705622 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:50.706295 22143 external_mini_cluster.cc:1428] Setting key 12d0a5667eb1a4dc95d11ab18772e9da
I20260620 14:58:50.706848 10167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:50.722990 10167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f: Bootstrap starting.
Jun 20 14:58:50 dist-test-slave-3mqx krb5kdc[10087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:58:50.727382 10167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:50.728858 10167 log.cc:824] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:50.732934 10167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f: No bootstrap required, opened a new log
I20260620 14:58:50.734552 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:50.708406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49219 (local address 127.0.0.1:45345)
0620 14:58:50.709248 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:50.709260 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:50.709303 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:58:50.711816 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:50.711841 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:50.712324 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:50.712744 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:50.714752 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:50.716219 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:50.718067 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:50.719250 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:50.719480 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:50.724866 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:50.724878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:50.724901 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:50.724984 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:50.729206 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:50.730497 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:50.730507 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:50.730516 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:50.730736 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:50.731298 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:50.731303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:50.731306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:50.732074 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:50.732421 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:50.732756 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 14:58:50.733123 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":393,"threads_started":1}
I20260620 14:58:50.739159 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:50.742413 10167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 14:58:50.742890 10167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:50.742995 10167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a09a2b3a3a4503bcc0daf0e0e7cd0f, State: Initialized, Role: FOLLOWER
I20260620 14:58:50.743760 10167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 14:58:50.744035 10167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:50.744189 10167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:50.744422 10167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:50.747226 10167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 14:58:50.747917 10167 leader_election.cc:302] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44a09a2b3a3a4503bcc0daf0e0e7cd0f; no voters:
I20260620 14:58:50.748709 10167 leader_election.cc:288] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:50.748816 10177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:50.750231 10177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [term 1 LEADER]: Becoming Leader. State: Replica: 44a09a2b3a3a4503bcc0daf0e0e7cd0f, State: Running, Role: LEADER
I20260620 14:58:50.750955 10177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } }
I20260620 14:58:50.751456 10167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:50.755177 10179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44a09a2b3a3a4503bcc0daf0e0e7cd0f. Latest consensus state: current_term: 1 leader_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } } }
I20260620 14:58:50.755812 10179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:50.755381 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44a09a2b3a3a4503bcc0daf0e0e7cd0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45345 } } }
I20260620 14:58:50.756513 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:50.761444 10182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:50.766286 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:50.744431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53711 (local address 127.0.0.1:45345)
0620 14:58:50.744639 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:50.744645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:50.745132 (+ 487us) server_negotiation.cc:407] Connection header received
0620 14:58:50.745278 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:50.745285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:50.745386 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:50.745524 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:50.746983 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:50.748163 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:50.749937 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:50.750633 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:50.750773 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:50.754586 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:50.754595 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:50.754600 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:50.754672 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:50.759924 (+ 5252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:50.761505 (+ 1581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:50.761513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:50.761517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:50.761701 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:50.762515 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:50.762523 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:50.762529 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:50.763193 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:50.765107 (+ 1914us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:50.765356 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:58:50.765650 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:58:50.767943 10182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:50.778857 10182 catalog_manager.cc:1347] Generated new cluster ID: 0eb704a1f5cb444fa4cbab84780dbd17
I20260620 14:58:50.779027 10182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:50.795083 10182 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:50.796433 10182 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:50.804972 10182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44a09a2b3a3a4503bcc0daf0e0e7cd0f: Generated new TSK 0
I20260620 14:58:50.806028 10182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:50.819622 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10100
I20260620 14:58:50.822149 10162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:50.916709 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10100
W20260620 14:58:50.934955 10085 connection.cc:535] client connection to 127.0.0.1:45345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45345): Connection reset by peer (error 104)
2026-06-20T14:58:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2134 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:51 dist-test-slave-3mqx krb5kdc[10199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:51 dist-test-slave-3mqx krb5kdc[10199](info): set up 2 sockets
Jun 20 14:58:51 dist-test-slave-3mqx krb5kdc[10199](info): commencing operation
krb5kdc: starting...
W20260620 14:58:52.083439 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:52 dist-test-slave-3mqx krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:58:52.346908 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:52.688424 10212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:52.688727 10212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:52.688906 10212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:52.697436 10212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:52.697577 10212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:52.697670 10212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:52.697757 10212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:52.697796 10212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:52.708444 10212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:52.709862 10212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:52.711246 10212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:52.723973 10217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:52.724047 10218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:52.724167 10220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:52.725158 10212 server_base.cc:1034] running on GCE node
I20260620 14:58:52.725977 10212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:52.728025 10212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:52.729285 10212 hybrid_clock.cc:648] HybridClock initialized: now 1781967532729240 us; error 44 us; skew 500 ppm
I20260620 14:58:52.729668 10212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:52 dist-test-slave-3mqx krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:52.733014 10212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:58:52.734577 10212 webserver.cc:460] Webserver started at http://127.0.0.1:35429/ using document root <none> and password file <none>
I20260620 14:58:52.735386 10212 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:52.735539 10212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:52.735865 10212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:52.739005 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "a6ac9293bc254375a4b332db7267ef59"
format_stamp: "Formatted at 2026-06-20 14:58:52 on dist-test-slave-3mqx"
server_key: "29d15e87837780196448e2587bc4b98c"
server_key_iv: "b96b83817f1be2eb39f6c770f48e361b"
server_key_version: "encryptionkey@0"
I20260620 14:58:52.739934 10212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "a6ac9293bc254375a4b332db7267ef59"
format_stamp: "Formatted at 2026-06-20 14:58:52 on dist-test-slave-3mqx"
server_key: "29d15e87837780196448e2587bc4b98c"
server_key_iv: "b96b83817f1be2eb39f6c770f48e361b"
server_key_version: "encryptionkey@0"
I20260620 14:58:52.745677 10212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:58:52.749783 10227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:52.751550 10212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:58:52.751766 10212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "a6ac9293bc254375a4b332db7267ef59"
format_stamp: "Formatted at 2026-06-20 14:58:52 on dist-test-slave-3mqx"
server_key: "29d15e87837780196448e2587bc4b98c"
server_key_iv: "b96b83817f1be2eb39f6c770f48e361b"
server_key_version: "encryptionkey@0"
I20260620 14:58:52.751996 10212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:52.776804 10212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:52.781194 10212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:52.781533 10212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:52.801220 10212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41629
I20260620 14:58:52.801282 10278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41629 every 8 connection(s)
I20260620 14:58:52.802712 10212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:52.806785 10279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:52.812470 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10212
I20260620 14:58:52.812708 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:52.813318 22143 external_mini_cluster.cc:1428] Setting key 03fb74ada95daa334e62c87251ee93a6
I20260620 14:58:52.820204 10279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59: Bootstrap starting.
I20260620 14:58:52.823731 10279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:52.825238 10279 log.cc:824] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:52.828307 10279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59: No bootstrap required, opened a new log
I20260620 14:58:52.836163 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:52.815457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58501 (local address 127.0.0.1:41629)
0620 14:58:52.816326 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:52.816339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:52.816388 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:58:52.818525 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:52.818552 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:52.819005 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:52.819500 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:52.820067 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:52.821445 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:52.822812 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:52.823575 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:52.823791 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:52.833264 (+ 9473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:52.833273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:52.833291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:52.833994 (+ 703us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:52.834101 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:52.834155 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:52.834376 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:52.834766 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 14:58:52.834951 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":410,"threads_started":1}
I20260620 14:58:52.837648 10279 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ac9293bc254375a4b332db7267ef59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41629 } }
I20260620 14:58:52.838135 10279 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:52.838287 10279 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ac9293bc254375a4b332db7267ef59, State: Initialized, Role: FOLLOWER
I20260620 14:58:52.839008 10279 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ac9293bc254375a4b332db7267ef59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41629 } }
I20260620 14:58:52.839391 10279 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:52.839553 10279 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:52.839756 10279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:52.840809 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:52.842401 10279 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ac9293bc254375a4b332db7267ef59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41629 } }
I20260620 14:58:52.842953 10279 leader_election.cc:302] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ac9293bc254375a4b332db7267ef59; no voters:
I20260620 14:58:52.843761 10279 leader_election.cc:288] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:52.843890 10285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:52.846331 10285 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [term 1 LEADER]: Becoming Leader. State: Replica: a6ac9293bc254375a4b332db7267ef59, State: Running, Role: LEADER
I20260620 14:58:52.846858 10279 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:52.847119 10285 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ac9293bc254375a4b332db7267ef59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41629 } }
I20260620 14:58:52.850732 10289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ac9293bc254375a4b332db7267ef59. Latest consensus state: current_term: 1 leader_uuid: "a6ac9293bc254375a4b332db7267ef59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ac9293bc254375a4b332db7267ef59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41629 } } }
I20260620 14:58:52.852126 10289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:52.851706 10290 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ac9293bc254375a4b332db7267ef59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ac9293bc254375a4b332db7267ef59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41629 } } }
I20260620 14:58:52.852784 10290 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:52.855536 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:52.845571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41927 (local address 127.0.0.1:41629)
0620 14:58:52.845794 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:52.845803 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:52.846428 (+ 625us) server_negotiation.cc:407] Connection header received
0620 14:58:52.846714 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:52.846719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:52.846805 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:52.846869 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:52.847687 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:52.848721 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:52.850803 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:52.851596 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:52.851776 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:52.853964 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:52.853974 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:52.853982 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:52.854018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:52.854131 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:52.854163 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:52.854309 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:52.854889 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 14:58:52.854965 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":66}
I20260620 14:58:52.860488 10296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:52.866137 10296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:52.875291 10296 catalog_manager.cc:1347] Generated new cluster ID: 465d16bb5d7b4b27ad63c584791e9f8f
I20260620 14:58:52.875438 10296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:52.890221 10296 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:52.891670 10296 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:52.905944 10296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ac9293bc254375a4b332db7267ef59: Generated new TSK 0
I20260620 14:58:52.906915 10296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:52.923791 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10212
W20260620 14:58:52.925450 10274 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58501
W20260620 14:58:52.926343 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:58:52.926561 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10212
W20260620 14:58:52.944435 10197 connection.cc:535] client connection to 127.0.0.1:41629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41629): Connection reset by peer (error 104)
2026-06-20T14:58:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2010 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:53 dist-test-slave-3mqx krb5kdc[10311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:53 dist-test-slave-3mqx krb5kdc[10311](info): set up 2 sockets
Jun 20 14:58:53 dist-test-slave-3mqx krb5kdc[10311](info): commencing operation
krb5kdc: starting...
W20260620 14:58:54.127498 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:54 dist-test-slave-3mqx krb5kdc[10311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:58:54.394515 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:54.740000 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:54.740326 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:54.740492 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:54.748350 10324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:54.748497 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:54.748544 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:54.748667 10324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:54.748733 10324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:54.758980 10324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:54.760412 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:54.761842 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:54.773950 10329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:54.773989 10332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:54.774017 10330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:54.774901 10324 server_base.cc:1034] running on GCE node
I20260620 14:58:54.775766 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:54.777336 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:54.778554 10324 hybrid_clock.cc:648] HybridClock initialized: now 1781967534778511 us; error 38 us; skew 500 ppm
I20260620 14:58:54.778965 10324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:54 dist-test-slave-3mqx krb5kdc[10311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:54.782393 10324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:58:54.784011 10324 webserver.cc:460] Webserver started at http://127.0.0.1:42373/ using document root <none> and password file <none>
I20260620 14:58:54.784696 10324 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:54.784838 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:54.785154 10324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:54.788296 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "44fd384115bb48f9a1fa6f0698947a28"
format_stamp: "Formatted at 2026-06-20 14:58:54 on dist-test-slave-3mqx"
server_key: "44d03c2c8c52140779e015d58becbd57"
server_key_iv: "fe8a8c244e5acc179e5bf2f6b1ec1336"
server_key_version: "encryptionkey@0"
I20260620 14:58:54.789105 10324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "44fd384115bb48f9a1fa6f0698947a28"
format_stamp: "Formatted at 2026-06-20 14:58:54 on dist-test-slave-3mqx"
server_key: "44d03c2c8c52140779e015d58becbd57"
server_key_iv: "fe8a8c244e5acc179e5bf2f6b1ec1336"
server_key_version: "encryptionkey@0"
I20260620 14:58:54.794476 10324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:58:54.798010 10339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:54.799522 10324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:58:54.799666 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "44fd384115bb48f9a1fa6f0698947a28"
format_stamp: "Formatted at 2026-06-20 14:58:54 on dist-test-slave-3mqx"
server_key: "44d03c2c8c52140779e015d58becbd57"
server_key_iv: "fe8a8c244e5acc179e5bf2f6b1ec1336"
server_key_version: "encryptionkey@0"
I20260620 14:58:54.799909 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:54.805789 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:54.810734 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:54.811163 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:54.832123 10324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37351
I20260620 14:58:54.832158 10390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37351 every 8 connection(s)
I20260620 14:58:54.833593 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:54.837889 10391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:54.842293 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10324
I20260620 14:58:54.842500 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:54.842948 22143 external_mini_cluster.cc:1428] Setting key 6efa1606a6783e2d53ca3fffa1c6977d
I20260620 14:58:54.852703 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:54.844981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56409 (local address 127.0.0.1:37351)
0620 14:58:54.845856 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:54.845874 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:54.845931 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:58:54.848191 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:54.848231 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:54.848811 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:54.849353 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:54.850497 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:54.850507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:54.850532 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:54.850557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:54.850633 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:54.850663 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:54.850915 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:54.851103 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:58:54.851320 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":328,"threads_started":1}
I20260620 14:58:54.853521 10391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28: Bootstrap starting.
I20260620 14:58:54.856650 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:54.857363 10391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:54.858464 10391 log.cc:824] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:54.863107 10391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28: No bootstrap required, opened a new log
I20260620 14:58:54.865348 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:54.862803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60733 (local address 127.0.0.1:37351)
0620 14:58:54.862990 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:54.862997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:54.863326 (+ 329us) server_negotiation.cc:407] Connection header received
0620 14:58:54.863510 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:54.863517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:54.863582 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:54.863728 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:54.864639 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:54.864645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:54.864650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:54.864670 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:54.864718 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:54.864740 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:54.864876 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:54.864936 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:58:54.865001 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:58:54.872674 10391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44fd384115bb48f9a1fa6f0698947a28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37351 } }
I20260620 14:58:54.873073 10391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:54.873157 10391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44fd384115bb48f9a1fa6f0698947a28, State: Initialized, Role: FOLLOWER
I20260620 14:58:54.873809 10391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44fd384115bb48f9a1fa6f0698947a28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37351 } }
I20260620 14:58:54.874050 10391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:54.874157 10391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:54.874300 10391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:54.876814 10391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44fd384115bb48f9a1fa6f0698947a28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37351 } }
I20260620 14:58:54.877337 10391 leader_election.cc:302] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44fd384115bb48f9a1fa6f0698947a28; no voters:
I20260620 14:58:54.878036 10391 leader_election.cc:288] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:54.878197 10401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:54.879333 10401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [term 1 LEADER]: Becoming Leader. State: Replica: 44fd384115bb48f9a1fa6f0698947a28, State: Running, Role: LEADER
I20260620 14:58:54.879869 10401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44fd384115bb48f9a1fa6f0698947a28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37351 } }
I20260620 14:58:54.880487 10391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:54.882843 10402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44fd384115bb48f9a1fa6f0698947a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44fd384115bb48f9a1fa6f0698947a28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37351 } } }
I20260620 14:58:54.882824 10403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44fd384115bb48f9a1fa6f0698947a28. Latest consensus state: current_term: 1 leader_uuid: "44fd384115bb48f9a1fa6f0698947a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44fd384115bb48f9a1fa6f0698947a28" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37351 } } }
I20260620 14:58:54.883394 10403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:54.883402 10402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:54.885015 10407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:54.891122 10407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:54.901182 10407 catalog_manager.cc:1347] Generated new cluster ID: 45463ff0e3bc441a81782e95dd2b020a
I20260620 14:58:54.901315 10407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:54.917193 10407 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:54.917685 10407 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:54.928556 10407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44fd384115bb48f9a1fa6f0698947a28: Generated new TSK 0
I20260620 14:58:54.929726 10407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:54.938383 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10324
W20260620 14:58:54.939743 10386 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56409
W20260620 14:58:54.940521 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10324: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:58:54.940670 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10324
2026-06-20T14:58:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:55 dist-test-slave-3mqx krb5kdc[10422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:55 dist-test-slave-3mqx krb5kdc[10422](info): set up 2 sockets
Jun 20 14:58:55 dist-test-slave-3mqx krb5kdc[10422](info): commencing operation
krb5kdc: starting...
W20260620 14:58:56.071568 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:56 dist-test-slave-3mqx krb5kdc[10422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:58:56.334399 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:56.686002 10435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:56.686345 10435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:56.686520 10435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:56.694175 10435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:56.694320 10435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:56.694408 10435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:56.694512 10435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:56.694585 10435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:56.705163 10435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:56.706631 10435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:56.708149 10435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:56.721004 10440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:56.721072 10441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:56.721051 10443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:56.722575 10435 server_base.cc:1034] running on GCE node
I20260620 14:58:56.723598 10435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:56.725276 10435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:56.726596 10435 hybrid_clock.cc:648] HybridClock initialized: now 1781967536726551 us; error 46 us; skew 500 ppm
I20260620 14:58:56.727141 10435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:56 dist-test-slave-3mqx krb5kdc[10422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:56.730741 10435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:58:56.732563 10435 webserver.cc:460] Webserver started at http://127.0.0.1:36619/ using document root <none> and password file <none>
I20260620 14:58:56.733292 10435 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:56.733402 10435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:56.733703 10435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:56.737174 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "3e0f15262cbe423c83da663a05c315dc"
format_stamp: "Formatted at 2026-06-20 14:58:56 on dist-test-slave-3mqx"
server_key: "672191b6d94cdf6b6bcc72ed51e5b928"
server_key_iv: "b339963836110143857d532d2b484caf"
server_key_version: "encryptionkey@0"
I20260620 14:58:56.737989 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "3e0f15262cbe423c83da663a05c315dc"
format_stamp: "Formatted at 2026-06-20 14:58:56 on dist-test-slave-3mqx"
server_key: "672191b6d94cdf6b6bcc72ed51e5b928"
server_key_iv: "b339963836110143857d532d2b484caf"
server_key_version: "encryptionkey@0"
I20260620 14:58:56.744216 10435 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 14:58:56.748719 10450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:56.750459 10435 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:58:56.750706 10435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "3e0f15262cbe423c83da663a05c315dc"
format_stamp: "Formatted at 2026-06-20 14:58:56 on dist-test-slave-3mqx"
server_key: "672191b6d94cdf6b6bcc72ed51e5b928"
server_key_iv: "b339963836110143857d532d2b484caf"
server_key_version: "encryptionkey@0"
I20260620 14:58:56.750963 10435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:56.773813 10435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:56.778757 10435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:56.779134 10435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:56.799010 10435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33595
I20260620 14:58:56.799028 10501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33595 every 8 connection(s)
I20260620 14:58:56.800568 10435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:56.801685 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10435
I20260620 14:58:56.801908 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:56.802454 22143 external_mini_cluster.cc:1428] Setting key 4d0bbb9cf366f54141e658c77bcf9302
I20260620 14:58:56.805766 10502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:58:56 dist-test-slave-3mqx krb5kdc[10422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:58:56.817510 10502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc: Bootstrap starting.
I20260620 14:58:56.821471 10502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:56.822613 10502 log.cc:824] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:56.826479 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:56.804425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45697 (local address 127.0.0.1:33595)
0620 14:58:56.805235 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:56.805252 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:56.805295 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:58:56.807205 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:56.807230 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:56.807657 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:56.808067 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:56.809747 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:56.811218 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:56.812572 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:56.813316 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:56.813533 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:56.817957 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:56.817965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:56.817982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:56.818032 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:56.821349 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:56.822229 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:56.822237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:56.822246 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:56.822431 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:56.822934 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:56.822938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:56.822940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:56.823659 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:56.823900 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:56.824544 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 14:58:56.825033 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":365,"threads_started":1}
I20260620 14:58:56.826473 10502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc: No bootstrap required, opened a new log
I20260620 14:58:56.830981 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:56.835737 10502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0f15262cbe423c83da663a05c315dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260620 14:58:56.836308 10502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:56.836452 10502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0f15262cbe423c83da663a05c315dc, State: Initialized, Role: FOLLOWER
I20260620 14:58:56.837186 10502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0f15262cbe423c83da663a05c315dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260620 14:58:56.837507 10502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:56.837657 10502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:56.837853 10502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:56.841157 10502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0f15262cbe423c83da663a05c315dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260620 14:58:56.841825 10502 leader_election.cc:302] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e0f15262cbe423c83da663a05c315dc; no voters:
I20260620 14:58:56.842551 10502 leader_election.cc:288] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:56.842653 10512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:56.843439 10512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [term 1 LEADER]: Becoming Leader. State: Replica: 3e0f15262cbe423c83da663a05c315dc, State: Running, Role: LEADER
I20260620 14:58:56.844287 10512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0f15262cbe423c83da663a05c315dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260620 14:58:56.847342 10502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:56.848415 10514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e0f15262cbe423c83da663a05c315dc. Latest consensus state: current_term: 1 leader_uuid: "3e0f15262cbe423c83da663a05c315dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0f15262cbe423c83da663a05c315dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260620 14:58:56.848464 10513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e0f15262cbe423c83da663a05c315dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e0f15262cbe423c83da663a05c315dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260620 14:58:56.848923 10513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:56.848928 10514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:56.852733 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:56.835599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41789 (local address 127.0.0.1:33595)
0620 14:58:56.835823 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:56.835831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:56.836212 (+ 381us) server_negotiation.cc:407] Connection header received
0620 14:58:56.836404 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:56.836409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:56.836525 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:56.836636 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:56.838859 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:56.839751 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:56.841614 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:56.842153 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:56.842317 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:56.845221 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:56.845233 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:56.845239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:58:56.845289 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:56.849103 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:56.850100 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:56.850105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:56.850109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:56.850336 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:58:56.850879 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:58:56.850883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:58:56.850887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:58:56.851313 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:56.851602 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:56.852112 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 14:58:56.852405 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:58:56.859545 10518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:56.866209 10518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:56.875487 10518 catalog_manager.cc:1347] Generated new cluster ID: 2d449b5b4dd34ce2ae02db6ad759461e
I20260620 14:58:56.875633 10518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:56.903828 10518 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:56.905937 10518 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:56.916791 10518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e0f15262cbe423c83da663a05c315dc: Generated new TSK 0
I20260620 14:58:56.917753 10518 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:56.925522 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10435
I20260620 14:58:56.926767 10497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:58:57.023290 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10435
2026-06-20T14:58:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:57 dist-test-slave-3mqx krb5kdc[10534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:57 dist-test-slave-3mqx krb5kdc[10534](info): set up 2 sockets
Jun 20 14:58:57 dist-test-slave-3mqx krb5kdc[10534](info): commencing operation
krb5kdc: starting...
W20260620 14:58:58.207998 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:58:58 dist-test-slave-3mqx krb5kdc[10534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:58:58.490541 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:58:58.848133 10547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:58:58.848526 10547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:58:58.848730 10547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:58:58.856218 10547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:58:58.856406 10547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:58:58.856494 10547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:58:58.856598 10547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:58:58.856673 10547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:58:58.867254 10547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:58:58.868719 10547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:58:58.870121 10547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:58:58.882858 10555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:58.882946 10552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:58:58.882915 10553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:58:58.884099 10547 server_base.cc:1034] running on GCE node
I20260620 14:58:58.884999 10547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:58:58.886698 10547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:58:58.887936 10547 hybrid_clock.cc:648] HybridClock initialized: now 1781967538887896 us; error 65 us; skew 500 ppm
I20260620 14:58:58.888355 10547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:58:58 dist-test-slave-3mqx krb5kdc[10534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:58:58.895284 10547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:58:58.897007 10547 webserver.cc:460] Webserver started at http://127.0.0.1:41031/ using document root <none> and password file <none>
I20260620 14:58:58.897789 10547 fs_manager.cc:359] Metadata directory not provided
I20260620 14:58:58.897949 10547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:58:58.898284 10547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:58:58.901625 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "90db3e29988549f2b261857afe172295"
format_stamp: "Formatted at 2026-06-20 14:58:58 on dist-test-slave-3mqx"
server_key: "a4bf598ab1f3c77db9819d09e29137c1"
server_key_iv: "17196b3fdeee708557f6e57fe8309a8d"
server_key_version: "encryptionkey@0"
I20260620 14:58:58.902467 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "90db3e29988549f2b261857afe172295"
format_stamp: "Formatted at 2026-06-20 14:58:58 on dist-test-slave-3mqx"
server_key: "a4bf598ab1f3c77db9819d09e29137c1"
server_key_iv: "17196b3fdeee708557f6e57fe8309a8d"
server_key_version: "encryptionkey@0"
I20260620 14:58:58.908480 10547 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260620 14:58:58.912513 10562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:58:58.914016 10547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:58:58.914234 10547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "90db3e29988549f2b261857afe172295"
format_stamp: "Formatted at 2026-06-20 14:58:58 on dist-test-slave-3mqx"
server_key: "a4bf598ab1f3c77db9819d09e29137c1"
server_key_iv: "17196b3fdeee708557f6e57fe8309a8d"
server_key_version: "encryptionkey@0"
I20260620 14:58:58.914463 10547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:58:58.944123 10547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:58:58.949240 10547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:58:58.949573 10547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:58:58.970144 10547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43215
I20260620 14:58:58.970167 10613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43215 every 8 connection(s)
I20260620 14:58:58.971728 10547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:58:58.975683 10614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:58:58.978401 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10547
I20260620 14:58:58.978592 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:58:58.979151 22143 external_mini_cluster.cc:1428] Setting key 8e9573a09bd9ed5793abb723c8bb1deb
I20260620 14:58:58.988241 10614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295: Bootstrap starting.
I20260620 14:58:58.991541 10614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295: Neither blocks nor log segments found. Creating new log.
I20260620 14:58:58.992851 10614 log.cc:824] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295: Log is configured to *not* fsync() on all Append() calls
I20260620 14:58:58.994025 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:58.981343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54019 (local address 127.0.0.1:43215)
0620 14:58:58.982241 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:58.982255 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:58.982304 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:58:58.984204 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:58.984231 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:58.984703 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:58.985183 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:58.985798 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:58.987250 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:58.988725 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:58.989420 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:58.989660 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:58.990854 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:58.990862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:58.990877 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:58.991538 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:58.991644 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:58.991696 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:58.991821 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:58.992141 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:58:58.992499 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":387,"threads_started":1}
I20260620 14:58:58.996285 10614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295: No bootstrap required, opened a new log
I20260620 14:58:58.998545 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:58:59.007192 10614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90db3e29988549f2b261857afe172295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260620 14:58:59.007869 10614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:58:59.008006 10614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90db3e29988549f2b261857afe172295, State: Initialized, Role: FOLLOWER
I20260620 14:58:59.008842 10614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90db3e29988549f2b261857afe172295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260620 14:58:59.009203 10614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:58:59.009387 10614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:58:59.009598 10614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:58:59.011837 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:58:59.004256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54363 (local address 127.0.0.1:43215)
0620 14:58:59.004475 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:58:59.004481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:58:59.004864 (+ 383us) server_negotiation.cc:407] Connection header received
0620 14:58:59.005045 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:58:59.005051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:58:59.005150 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:58:59.005278 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:58:59.005968 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:59.007003 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:59.008837 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:58:59.009614 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:58:59.009796 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:58:59.011009 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:58:59.011017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:58:59.011022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:58:59.011047 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:58:59.011136 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:58:59.011161 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:58:59.011285 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:58:59.011370 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:58:59.011483 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:58:59.012929 10614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90db3e29988549f2b261857afe172295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260620 14:58:59.013621 10614 leader_election.cc:302] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90db3e29988549f2b261857afe172295; no voters:
I20260620 14:58:59.014283 10614 leader_election.cc:288] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:58:59.014446 10624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:58:59.015301 10624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [term 1 LEADER]: Becoming Leader. State: Replica: 90db3e29988549f2b261857afe172295, State: Running, Role: LEADER
I20260620 14:58:59.015925 10624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90db3e29988549f2b261857afe172295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } }
I20260620 14:58:59.016757 10614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:58:59.019099 10626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90db3e29988549f2b261857afe172295. Latest consensus state: current_term: 1 leader_uuid: "90db3e29988549f2b261857afe172295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90db3e29988549f2b261857afe172295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } } }
I20260620 14:58:59.019619 10626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:59.020277 10625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90db3e29988549f2b261857afe172295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90db3e29988549f2b261857afe172295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43215 } } }
I20260620 14:58:59.020767 10625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295 [sys.catalog]: This master's current role is: LEADER
I20260620 14:58:59.022964 10629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:58:59.030519 10629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:58:59.040903 10629 catalog_manager.cc:1347] Generated new cluster ID: 8b37f1cd685447c3bc46d43da29719c6
I20260620 14:58:59.041057 10629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:58:59.057255 10629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:58:59.059387 10629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:58:59.070493 10629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90db3e29988549f2b261857afe172295: Generated new TSK 0
I20260620 14:58:59.071878 10629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:58:59.085347 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10547
W20260620 14:58:59.086560 10609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54019
W20260620 14:58:59.087488 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:58:59.087656 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10547
2026-06-20T14:58:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:58:59 dist-test-slave-3mqx krb5kdc[10645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:58:59 dist-test-slave-3mqx krb5kdc[10645](info): set up 2 sockets
Jun 20 14:58:59 dist-test-slave-3mqx krb5kdc[10645](info): commencing operation
krb5kdc: starting...
W20260620 14:59:00.215533 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:00 dist-test-slave-3mqx krb5kdc[10645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:59:00.490203 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:00.852598 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:00.853018 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:00.853199 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:00.861353 10658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:00.861578 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:00.861692 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:00.861812 10658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:00.861901 10658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:00.872960 10658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:00.874475 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:00.875993 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:00.889420 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:00.889503 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:00.889813 10666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:00.890800 10658 server_base.cc:1034] running on GCE node
I20260620 14:59:00.891716 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:00.893323 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:00.894569 10658 hybrid_clock.cc:648] HybridClock initialized: now 1781967540894513 us; error 55 us; skew 500 ppm
I20260620 14:59:00.894949 10658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:00 dist-test-slave-3mqx krb5kdc[10645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:00.898228 10658 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:59:00.899986 10658 webserver.cc:460] Webserver started at http://127.0.0.1:38411/ using document root <none> and password file <none>
I20260620 14:59:00.900727 10658 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:00.900884 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:00.901208 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:00.904507 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "a398dd6463654d48a20c1fc34f87d943"
format_stamp: "Formatted at 2026-06-20 14:59:00 on dist-test-slave-3mqx"
server_key: "06ab6df10274f7cd9c43a6029f051aaa"
server_key_iv: "276e4de6a6d8c005a0ef5f48cdd45e97"
server_key_version: "encryptionkey@0"
I20260620 14:59:00.905341 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "a398dd6463654d48a20c1fc34f87d943"
format_stamp: "Formatted at 2026-06-20 14:59:00 on dist-test-slave-3mqx"
server_key: "06ab6df10274f7cd9c43a6029f051aaa"
server_key_iv: "276e4de6a6d8c005a0ef5f48cdd45e97"
server_key_version: "encryptionkey@0"
I20260620 14:59:00.910868 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 14:59:00.914696 10673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:00.916179 10658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:59:00.916383 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "a398dd6463654d48a20c1fc34f87d943"
format_stamp: "Formatted at 2026-06-20 14:59:00 on dist-test-slave-3mqx"
server_key: "06ab6df10274f7cd9c43a6029f051aaa"
server_key_iv: "276e4de6a6d8c005a0ef5f48cdd45e97"
server_key_version: "encryptionkey@0"
I20260620 14:59:00.916607 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:00.922922 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:00.927518 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:00.927850 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:00.947037 10658 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39265
I20260620 14:59:00.947103 10724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39265 every 8 connection(s)
I20260620 14:59:00.948553 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:00.952539 10725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:00.958134 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10658
I20260620 14:59:00.958330 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:00.958863 22143 external_mini_cluster.cc:1428] Setting key 2c8147db285edde7b6698c28b52f3080
I20260620 14:59:00.964632 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943: Bootstrap starting.
I20260620 14:59:00.967294 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:00.960896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44995 (local address 127.0.0.1:39265)
0620 14:59:00.961715 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:00.961727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:00.961772 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:59:00.963631 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:00.963664 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:00.964041 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:00.964439 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:00.965551 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:00.965558 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:00.965572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:00.965585 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:00.965653 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:00.965673 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:00.965905 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:00.966048 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:59:00.966242 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":339,"threads_started":1}
I20260620 14:59:00.968501 10725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:00.969683 10725 log.cc:824] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:00.971439 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:00.972846 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943: No bootstrap required, opened a new log
I20260620 14:59:00.978487 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:00.976027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40953 (local address 127.0.0.1:39265)
0620 14:59:00.976232 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:00.976238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:00.976545 (+ 307us) server_negotiation.cc:407] Connection header received
0620 14:59:00.976694 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:00.976700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:00.976763 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:00.976872 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:00.977820 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:00.977827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:00.977832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:00.977851 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:00.977898 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:00.977919 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:00.978042 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:00.978108 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:59:00.978173 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:59:00.981812 10725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a398dd6463654d48a20c1fc34f87d943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260620 14:59:00.982224 10725 raft_consensus.cc:374] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:00.982342 10725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a398dd6463654d48a20c1fc34f87d943, State: Initialized, Role: FOLLOWER
I20260620 14:59:00.982921 10725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a398dd6463654d48a20c1fc34f87d943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260620 14:59:00.983232 10725 raft_consensus.cc:388] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:00.983340 10725 raft_consensus.cc:482] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:00.983522 10725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:00.986239 10725 raft_consensus.cc:504] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a398dd6463654d48a20c1fc34f87d943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260620 14:59:00.986821 10725 leader_election.cc:302] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a398dd6463654d48a20c1fc34f87d943; no voters:
I20260620 14:59:00.987618 10725 leader_election.cc:288] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:00.987813 10735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:00.988552 10735 raft_consensus.cc:686] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [term 1 LEADER]: Becoming Leader. State: Replica: a398dd6463654d48a20c1fc34f87d943, State: Running, Role: LEADER
I20260620 14:59:00.989279 10735 consensus_queue.cc:237] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a398dd6463654d48a20c1fc34f87d943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } }
I20260620 14:59:00.990727 10725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:00.992610 10736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a398dd6463654d48a20c1fc34f87d943. Latest consensus state: current_term: 1 leader_uuid: "a398dd6463654d48a20c1fc34f87d943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a398dd6463654d48a20c1fc34f87d943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } } }
I20260620 14:59:00.992694 10737 sys_catalog.cc:455] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a398dd6463654d48a20c1fc34f87d943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a398dd6463654d48a20c1fc34f87d943" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39265 } } }
I20260620 14:59:00.993098 10737 sys_catalog.cc:458] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:00.993103 10736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:00.995918 10740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:01.000736 10740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:01.011173 10740 catalog_manager.cc:1347] Generated new cluster ID: 7e90665f6f324a86b058814fea4595df
I20260620 14:59:01.011315 10740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:01.054416 10740 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:01.054970 10740 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:01.065181 10740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a398dd6463654d48a20c1fc34f87d943: Generated new TSK 0
I20260620 14:59:01.066375 10740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:01.074517 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10658
W20260620 14:59:01.075737 10720 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44995
W20260620 14:59:01.076566 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:01.076774 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10658
2026-06-20T14:59:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:01 dist-test-slave-3mqx krb5kdc[10756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:01 dist-test-slave-3mqx krb5kdc[10756](info): set up 2 sockets
Jun 20 14:59:01 dist-test-slave-3mqx krb5kdc[10756](info): commencing operation
krb5kdc: starting...
W20260620 14:59:02.227517 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:02 dist-test-slave-3mqx krb5kdc[10756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:59:02.498157 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:02.841370 10769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:02.841780 10769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:02.841948 10769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:02.849448 10769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:02.849637 10769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:02.849756 10769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:02.849864 10769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:02.849941 10769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:02.860595 10769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:02.862047 10769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:02.863637 10769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:02.876348 10774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:02.876323 10777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:02.876272 10775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:02.877377 10769 server_base.cc:1034] running on GCE node
I20260620 14:59:02.878319 10769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:02.880051 10769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:02.881335 10769 hybrid_clock.cc:648] HybridClock initialized: now 1781967542881280 us; error 65 us; skew 500 ppm
I20260620 14:59:02.881771 10769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:02 dist-test-slave-3mqx krb5kdc[10756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:02.885366 10769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:59:02.887037 10769 webserver.cc:460] Webserver started at http://127.0.0.1:43565/ using document root <none> and password file <none>
I20260620 14:59:02.887840 10769 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:02.888006 10769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:02.888332 10769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:02.891532 10769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "8e75e3139bdc45479dbc8be175f72002"
format_stamp: "Formatted at 2026-06-20 14:59:02 on dist-test-slave-3mqx"
server_key: "519c9fa7d2c21e4149fbfebf95a83281"
server_key_iv: "490185a985d9218687345fc71f7aeab2"
server_key_version: "encryptionkey@0"
I20260620 14:59:02.892364 10769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "8e75e3139bdc45479dbc8be175f72002"
format_stamp: "Formatted at 2026-06-20 14:59:02 on dist-test-slave-3mqx"
server_key: "519c9fa7d2c21e4149fbfebf95a83281"
server_key_iv: "490185a985d9218687345fc71f7aeab2"
server_key_version: "encryptionkey@0"
I20260620 14:59:02.897981 10769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:59:02.901784 10784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:02.903335 10769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:59:02.903529 10769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "8e75e3139bdc45479dbc8be175f72002"
format_stamp: "Formatted at 2026-06-20 14:59:02 on dist-test-slave-3mqx"
server_key: "519c9fa7d2c21e4149fbfebf95a83281"
server_key_iv: "490185a985d9218687345fc71f7aeab2"
server_key_version: "encryptionkey@0"
I20260620 14:59:02.903738 10769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:02.920198 10769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:02.924921 10769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:02.925246 10769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:02.945595 10769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44515
I20260620 14:59:02.945616 10835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44515 every 8 connection(s)
I20260620 14:59:02.947095 10769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:02.951167 10836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:02.955610 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10769
I20260620 14:59:02.955809 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:02.956348 22143 external_mini_cluster.cc:1428] Setting key 7bb6b58df8e8346b63d1d495bf8218ab
I20260620 14:59:02.963076 10836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002: Bootstrap starting.
I20260620 14:59:02.966804 10836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:02.968113 10836 log.cc:824] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:02.971376 10836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002: No bootstrap required, opened a new log
Jun 20 14:59:02 dist-test-slave-3mqx krb5kdc[10756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:59:02.982290 10836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75e3139bdc45479dbc8be175f72002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 14:59:02.982767 10836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:02.982882 10836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e75e3139bdc45479dbc8be175f72002, State: Initialized, Role: FOLLOWER
I20260620 14:59:02.983420 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:02.958340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:44601 (local address 127.0.0.1:44515)
0620 14:59:02.959209 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:02.959220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:02.959261 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:59:02.961570 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:02.961596 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:02.962030 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:02.962369 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:02.964279 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:02.965939 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:02.967842 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:02.968861 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:02.969111 (+ 250us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:02.973759 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:02.973775 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:02.973796 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:02.973841 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:02.978104 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:02.979163 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:02.979172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:02.979182 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:02.979379 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:02.979893 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:02.979898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:02.979902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:02.980692 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:02.981021 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:02.981509 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 14:59:02.981874 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260620 14:59:02.983641 10836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75e3139bdc45479dbc8be175f72002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 14:59:02.983944 10836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:02.984118 10836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:02.984314 10836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:02.986814 10836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75e3139bdc45479dbc8be175f72002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 14:59:02.987452 10836 leader_election.cc:302] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e75e3139bdc45479dbc8be175f72002; no voters:
I20260620 14:59:02.988103 10836 leader_election.cc:288] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:02.988268 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:02.988258 10841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:02.990288 10836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:02.991381 10841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [term 1 LEADER]: Becoming Leader. State: Replica: 8e75e3139bdc45479dbc8be175f72002, State: Running, Role: LEADER
I20260620 14:59:02.992200 10841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75e3139bdc45479dbc8be175f72002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260620 14:59:02.996619 10843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e75e3139bdc45479dbc8be175f72002. Latest consensus state: current_term: 1 leader_uuid: "8e75e3139bdc45479dbc8be175f72002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75e3139bdc45479dbc8be175f72002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } } }
I20260620 14:59:02.997136 10843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:02.997828 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e75e3139bdc45479dbc8be175f72002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e75e3139bdc45479dbc8be175f72002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } } }
I20260620 14:59:02.998112 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:03.000648 10853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:03.008412 10853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:03.026716 10863 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:59:03.027730 10853 catalog_manager.cc:1347] Generated new cluster ID: 886d21d9ce1b4a4e9f399969c476cbf7
I20260620 14:59:03.027858 10853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:03.036235 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:03.002847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:38033 (local address 127.0.0.1:44515)
0620 14:59:03.003148 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:03.003157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:03.003718 (+ 561us) server_negotiation.cc:407] Connection header received
0620 14:59:03.003936 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:03.003941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:03.004021 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:03.004140 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:03.006231 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:03.008648 (+ 2417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:03.015273 (+ 6625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:03.016452 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:03.016604 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:03.027880 (+ 11276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:03.027890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:03.027897 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:03.027954 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:03.032083 (+ 4129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:03.033308 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:03.033317 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:03.033321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:03.033480 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:03.034137 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:03.034142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:03.034146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:03.034597 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:03.034866 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:03.035634 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 14:59:03.035825 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:59:03.063845 10853 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:03.065673 10853 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:03.075133 10853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e75e3139bdc45479dbc8be175f72002: Generated new TSK 0
I20260620 14:59:03.076313 10853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:03.086803 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10769
I20260620 14:59:03.088642 10831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:03.184835 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10769
W20260620 14:59:03.202613 10754 connection.cc:535] client connection to 127.0.0.1:44515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44515): Connection reset by peer (error 104)
2026-06-20T14:59:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2108 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:03 dist-test-slave-3mqx krb5kdc[10868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:03 dist-test-slave-3mqx krb5kdc[10868](info): set up 2 sockets
Jun 20 14:59:03 dist-test-slave-3mqx krb5kdc[10868](info): commencing operation
krb5kdc: starting...
W20260620 14:59:04.299572 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:04 dist-test-slave-3mqx krb5kdc[10868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:59:04.587672 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:04.935640 10881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:04.935945 10881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:04.936129 10881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:04.943890 10881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:04.944013 10881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:04.944083 10881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:04.944204 10881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:04.944269 10881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:04.954720 10881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:04.956264 10881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:04.957664 10881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:04.971136 10886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:04.971136 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:04.971143 10887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:04.972256 10881 server_base.cc:1034] running on GCE node
I20260620 14:59:04.973153 10881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:04.974898 10881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:04.976151 10881 hybrid_clock.cc:648] HybridClock initialized: now 1781967544976102 us; error 55 us; skew 500 ppm
I20260620 14:59:04.976534 10881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:04 dist-test-slave-3mqx krb5kdc[10868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:04.985309 10881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:59:04.987336 10881 webserver.cc:460] Webserver started at http://127.0.0.1:35399/ using document root <none> and password file <none>
I20260620 14:59:04.988157 10881 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:04.988317 10881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:04.988682 10881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:04.992079 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "993334a9beff4806b838b87c4c90bf62"
format_stamp: "Formatted at 2026-06-20 14:59:04 on dist-test-slave-3mqx"
server_key: "a9716f108e2ca54668850af2e2051a32"
server_key_iv: "2e36e255921a2335318abefcd403300b"
server_key_version: "encryptionkey@0"
I20260620 14:59:04.992918 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "993334a9beff4806b838b87c4c90bf62"
format_stamp: "Formatted at 2026-06-20 14:59:04 on dist-test-slave-3mqx"
server_key: "a9716f108e2ca54668850af2e2051a32"
server_key_iv: "2e36e255921a2335318abefcd403300b"
server_key_version: "encryptionkey@0"
I20260620 14:59:04.999270 10881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:59:05.003566 10896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:05.005401 10881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:59:05.005625 10881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "993334a9beff4806b838b87c4c90bf62"
format_stamp: "Formatted at 2026-06-20 14:59:04 on dist-test-slave-3mqx"
server_key: "a9716f108e2ca54668850af2e2051a32"
server_key_iv: "2e36e255921a2335318abefcd403300b"
server_key_version: "encryptionkey@0"
I20260620 14:59:05.005853 10881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:05.028438 10881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:05.033423 10881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:05.033740 10881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:05.053406 10881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39845
I20260620 14:59:05.053512 10947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39845 every 8 connection(s)
I20260620 14:59:05.054906 10881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:05.056226 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10881
I20260620 14:59:05.056423 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:05.056866 22143 external_mini_cluster.cc:1428] Setting key 835b453aa4068f6c42af20d8c82f3018
I20260620 14:59:05.060309 10948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:05.070894 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:05.058880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:51239 (local address 127.0.0.1:39845)
0620 14:59:05.059804 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:05.059816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:05.059858 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:59:05.061635 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:05.061660 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:05.062068 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:05.062486 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:05.063125 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:05.064534 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:05.065927 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:05.066742 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:05.066949 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:05.068465 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:05.068475 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:05.068491 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:05.069057 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:05.069143 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:05.069177 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:05.069372 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:05.069618 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 14:59:05.069804 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":422,"threads_started":1}
I20260620 14:59:05.073102 10948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62: Bootstrap starting.
I20260620 14:59:05.075207 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:05.077015 10948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:05.078339 10948 log.cc:824] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:05.082037 10948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62: No bootstrap required, opened a new log
I20260620 14:59:05.088112 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:05.080165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:50223 (local address 127.0.0.1:39845)
0620 14:59:05.080362 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:05.080369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:05.080852 (+ 483us) server_negotiation.cc:407] Connection header received
0620 14:59:05.081045 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:05.081051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:05.081137 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:05.081269 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:05.082026 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:05.083277 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:05.084796 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:05.085557 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:05.085697 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:05.087151 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:05.087158 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:05.087163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:05.087190 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:05.087244 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:05.087272 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:05.087423 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:05.087675 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:59:05.087738 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:59:05.091812 10948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993334a9beff4806b838b87c4c90bf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 14:59:05.092242 10948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:05.092350 10948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993334a9beff4806b838b87c4c90bf62, State: Initialized, Role: FOLLOWER
I20260620 14:59:05.093091 10948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993334a9beff4806b838b87c4c90bf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 14:59:05.093406 10948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:05.093560 10948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:05.093763 10948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:05.096154 10948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993334a9beff4806b838b87c4c90bf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 14:59:05.096755 10948 leader_election.cc:302] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993334a9beff4806b838b87c4c90bf62; no voters:
I20260620 14:59:05.097427 10948 leader_election.cc:288] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:05.097684 10958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:05.099186 10958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [term 1 LEADER]: Becoming Leader. State: Replica: 993334a9beff4806b838b87c4c90bf62, State: Running, Role: LEADER
I20260620 14:59:05.099865 10948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:05.099864 10958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993334a9beff4806b838b87c4c90bf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 14:59:05.103031 10960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993334a9beff4806b838b87c4c90bf62. Latest consensus state: current_term: 1 leader_uuid: "993334a9beff4806b838b87c4c90bf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993334a9beff4806b838b87c4c90bf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } } }
I20260620 14:59:05.103560 10960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:05.103434 10959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "993334a9beff4806b838b87c4c90bf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993334a9beff4806b838b87c4c90bf62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } } }
I20260620 14:59:05.103807 10959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:05.106591 10966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:05.111738 10966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:05.122056 10966 catalog_manager.cc:1347] Generated new cluster ID: 32b7d3e20e9a412dae57a8b0436f0c62
I20260620 14:59:05.122228 10966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:05.142146 10966 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:05.143725 10966 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:05.160241 10966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 993334a9beff4806b838b87c4c90bf62: Generated new TSK 0
I20260620 14:59:05.161700 10966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:05.181923 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10881
W20260620 14:59:05.184089 10943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:51239
W20260620 14:59:05.184988 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10881: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:05.185142 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10881
W20260620 14:59:05.202627 10866 connection.cc:535] client connection to 127.0.0.1:39845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39845): Connection reset by peer (error 104)
2026-06-20T14:59:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:05 dist-test-slave-3mqx krb5kdc[10979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:05 dist-test-slave-3mqx krb5kdc[10979](info): set up 2 sockets
Jun 20 14:59:05 dist-test-slave-3mqx krb5kdc[10979](info): commencing operation
krb5kdc: starting...
W20260620 14:59:06.419729 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:06 dist-test-slave-3mqx krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:59:06.691879 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:07.044391 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:07.044827 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:07.045006 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:07.052717 10992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:07.052925 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:07.053041 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:07.053158 10992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:07.053248 10992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:07.064014 10992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:07.065495 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:07.066991 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:07.080181 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:07.080171 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:07.080277 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:07.081220 10992 server_base.cc:1034] running on GCE node
I20260620 14:59:07.082198 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:07.084277 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:07.085565 10992 hybrid_clock.cc:648] HybridClock initialized: now 1781967547085492 us; error 85 us; skew 500 ppm
I20260620 14:59:07.085971 10992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:07 dist-test-slave-3mqx krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:07.089757 10992 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:59:07.091683 10992 webserver.cc:460] Webserver started at http://127.0.0.1:33927/ using document root <none> and password file <none>
I20260620 14:59:07.092515 10992 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:07.092689 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:07.093053 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:07.096730 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "f10bb167c05d4f319a5b36502b003350"
format_stamp: "Formatted at 2026-06-20 14:59:07 on dist-test-slave-3mqx"
server_key: "e5a5f40ef3c5cc4e8f5cdc267248e584"
server_key_iv: "af490021e821124d67e18517e64ec55b"
server_key_version: "encryptionkey@0"
I20260620 14:59:07.097734 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "f10bb167c05d4f319a5b36502b003350"
format_stamp: "Formatted at 2026-06-20 14:59:07 on dist-test-slave-3mqx"
server_key: "e5a5f40ef3c5cc4e8f5cdc267248e584"
server_key_iv: "af490021e821124d67e18517e64ec55b"
server_key_version: "encryptionkey@0"
I20260620 14:59:07.104082 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 14:59:07.108412 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:07.109958 10992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260620 14:59:07.110184 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "f10bb167c05d4f319a5b36502b003350"
format_stamp: "Formatted at 2026-06-20 14:59:07 on dist-test-slave-3mqx"
server_key: "e5a5f40ef3c5cc4e8f5cdc267248e584"
server_key_iv: "af490021e821124d67e18517e64ec55b"
server_key_version: "encryptionkey@0"
I20260620 14:59:07.110448 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:07.116832 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:07.121906 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:07.122267 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:07.142014 10992 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34767
I20260620 14:59:07.142059 11058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34767 every 8 connection(s)
I20260620 14:59:07.143606 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:07.148272 11059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:07.149207 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 10992
I20260620 14:59:07.149462 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:07.150182 22143 external_mini_cluster.cc:1428] Setting key cf8fde24d9efe664a576f60c5862cfae
I20260620 14:59:07.160074 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:07.152331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:37389 (local address 127.0.0.1:34767)
0620 14:59:07.153305 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:07.153318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:07.153365 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:59:07.156095 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:07.156128 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:07.156582 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:07.156978 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:07.158292 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:07.158299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:07.158313 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:07.158328 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:07.158391 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:07.158415 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:07.158654 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:07.158797 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:59:07.159016 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":418,"threads_started":1}
I20260620 14:59:07.164448 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:07.165228 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350: Bootstrap starting.
I20260620 14:59:07.169589 11059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:07.171437 11059 log.cc:824] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:07.172076 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:07.169135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:58681 (local address 127.0.0.1:34767)
0620 14:59:07.169295 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:07.169299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:07.169830 (+ 531us) server_negotiation.cc:407] Connection header received
0620 14:59:07.170074 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:07.170078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:07.170138 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:07.170236 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:07.171386 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:07.171390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:07.171393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:07.171408 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:07.171451 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:07.171467 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:07.171559 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:07.171741 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 14:59:07.171793 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:59:07.175102 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350: No bootstrap required, opened a new log
I20260620 14:59:07.184235 11059 raft_consensus.cc:348] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10bb167c05d4f319a5b36502b003350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260620 14:59:07.184733 11059 raft_consensus.cc:374] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:07.184831 11059 raft_consensus.cc:729] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10bb167c05d4f319a5b36502b003350, State: Initialized, Role: FOLLOWER
I20260620 14:59:07.185475 11059 consensus_queue.cc:260] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10bb167c05d4f319a5b36502b003350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260620 14:59:07.185737 11059 raft_consensus.cc:388] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:07.185948 11059 raft_consensus.cc:482] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:07.186169 11059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:07.189206 11059 raft_consensus.cc:504] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10bb167c05d4f319a5b36502b003350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260620 14:59:07.189905 11059 leader_election.cc:302] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f10bb167c05d4f319a5b36502b003350; no voters:
I20260620 14:59:07.190675 11059 leader_election.cc:288] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:07.190861 11069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:07.191728 11069 raft_consensus.cc:686] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [term 1 LEADER]: Becoming Leader. State: Replica: f10bb167c05d4f319a5b36502b003350, State: Running, Role: LEADER
I20260620 14:59:07.192409 11069 consensus_queue.cc:237] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10bb167c05d4f319a5b36502b003350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } }
I20260620 14:59:07.193269 11059 sys_catalog.cc:564] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:07.195617 11070 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f10bb167c05d4f319a5b36502b003350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10bb167c05d4f319a5b36502b003350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } } }
I20260620 14:59:07.196261 11070 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:07.197860 11071 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f10bb167c05d4f319a5b36502b003350. Latest consensus state: current_term: 1 leader_uuid: "f10bb167c05d4f319a5b36502b003350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10bb167c05d4f319a5b36502b003350" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34767 } } }
I20260620 14:59:07.198143 11071 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:07.198932 11074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:07.205902 11074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:07.216661 11074 catalog_manager.cc:1347] Generated new cluster ID: fe4c9d42ecde43f3b82a701314db4442
I20260620 14:59:07.216822 11074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:07.262247 11074 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:07.262773 11074 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:07.271700 11074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f10bb167c05d4f319a5b36502b003350: Generated new TSK 0
I20260620 14:59:07.272670 11074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:07.284628 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10992
W20260620 14:59:07.285826 11054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:37389
W20260620 14:59:07.286656 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 10992: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:07.286877 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 10992
2026-06-20T14:59:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2101 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:07 dist-test-slave-3mqx krb5kdc[11090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:07 dist-test-slave-3mqx krb5kdc[11090](info): set up 2 sockets
Jun 20 14:59:07 dist-test-slave-3mqx krb5kdc[11090](info): commencing operation
krb5kdc: starting...
W20260620 14:59:08.423679 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:08 dist-test-slave-3mqx krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:59:08.706924 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:09.060868 11103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:09.061198 11103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:09.061379 11103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:09.069118 11103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:09.069258 11103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:09.069361 11103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:09.069482 11103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:09.069546 11103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:09.080410 11103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:09.081837 11103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:09.083280 11103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:09.099416 11111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:09.099327 11108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:09.099385 11109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:09.100427 11103 server_base.cc:1034] running on GCE node
I20260620 14:59:09.101359 11103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:09.103032 11103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:09.104373 11103 hybrid_clock.cc:648] HybridClock initialized: now 1781967549104269 us; error 100 us; skew 500 ppm
I20260620 14:59:09.104815 11103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:09 dist-test-slave-3mqx krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:09.108294 11103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:59:09.109994 11103 webserver.cc:460] Webserver started at http://127.0.0.1:33329/ using document root <none> and password file <none>
I20260620 14:59:09.110693 11103 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:09.110801 11103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:09.111107 11103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:09.114259 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "ef30cc846e244f6b8aedb902590d89be"
format_stamp: "Formatted at 2026-06-20 14:59:09 on dist-test-slave-3mqx"
server_key: "cd819fc8277fb9625df3bd47ae1279ec"
server_key_iv: "9d8432c7d27e4949f60b6b7338477215"
server_key_version: "encryptionkey@0"
I20260620 14:59:09.115118 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "ef30cc846e244f6b8aedb902590d89be"
format_stamp: "Formatted at 2026-06-20 14:59:09 on dist-test-slave-3mqx"
server_key: "cd819fc8277fb9625df3bd47ae1279ec"
server_key_iv: "9d8432c7d27e4949f60b6b7338477215"
server_key_version: "encryptionkey@0"
I20260620 14:59:09.120807 11103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 14:59:09.124881 11118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:09.126538 11103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:59:09.126715 11103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "ef30cc846e244f6b8aedb902590d89be"
format_stamp: "Formatted at 2026-06-20 14:59:09 on dist-test-slave-3mqx"
server_key: "cd819fc8277fb9625df3bd47ae1279ec"
server_key_iv: "9d8432c7d27e4949f60b6b7338477215"
server_key_version: "encryptionkey@0"
I20260620 14:59:09.126902 11103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:09.146389 11103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:09.151675 11103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:09.151949 11103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:09.173116 11103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39311
I20260620 14:59:09.173148 11169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39311 every 8 connection(s)
I20260620 14:59:09.174669 11103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:09.177861 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11103
I20260620 14:59:09.178081 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:09.178676 22143 external_mini_cluster.cc:1428] Setting key e7abb5e20d55934877d9976d843853c6
I20260620 14:59:09.179798 11170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:09.192705 11170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be: Bootstrap starting.
Jun 20 14:59:09 dist-test-slave-3mqx krb5kdc[11090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:59:09.196333 11170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:09.197481 11170 log.cc:824] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:09.201320 11170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be: No bootstrap required, opened a new log
I20260620 14:59:09.202675 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:09.180815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:37045 (local address 127.0.0.1:39311)
0620 14:59:09.181643 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:09.181656 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:09.181697 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:59:09.183434 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:09.183459 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:09.183915 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:09.184353 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:09.185915 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:09.187393 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:09.188733 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:09.189462 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:09.189689 (+ 227us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:09.194014 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:09.194023 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:09.194039 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:09.194088 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:09.197489 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:09.198570 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:09.198578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:09.198584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:09.198787 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:09.199386 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:09.199391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:09.199394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:09.200095 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:09.200437 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:09.200880 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 14:59:09.201236 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":364,"threads_started":1}
I20260620 14:59:09.207190 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:09.211511 11170 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef30cc846e244f6b8aedb902590d89be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 14:59:09.212080 11170 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:09.212195 11170 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef30cc846e244f6b8aedb902590d89be, State: Initialized, Role: FOLLOWER
I20260620 14:59:09.212996 11170 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef30cc846e244f6b8aedb902590d89be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 14:59:09.213330 11170 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:09.213478 11170 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:09.213692 11170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:09.216261 11170 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef30cc846e244f6b8aedb902590d89be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 14:59:09.216817 11170 leader_election.cc:302] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef30cc846e244f6b8aedb902590d89be; no voters:
I20260620 14:59:09.217471 11170 leader_election.cc:288] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:09.217661 11180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:09.218904 11180 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [term 1 LEADER]: Becoming Leader. State: Replica: ef30cc846e244f6b8aedb902590d89be, State: Running, Role: LEADER
I20260620 14:59:09.219645 11180 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef30cc846e244f6b8aedb902590d89be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 14:59:09.220041 11170 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:09.224344 11181 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef30cc846e244f6b8aedb902590d89be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef30cc846e244f6b8aedb902590d89be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } } }
I20260620 14:59:09.224861 11181 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:09.226280 11182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef30cc846e244f6b8aedb902590d89be. Latest consensus state: current_term: 1 leader_uuid: "ef30cc846e244f6b8aedb902590d89be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef30cc846e244f6b8aedb902590d89be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } } }
I20260620 14:59:09.226733 11182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:09.228714 11186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:09.233433 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:09.212874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:52523 (local address 127.0.0.1:39311)
0620 14:59:09.213085 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:09.213093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:09.213629 (+ 536us) server_negotiation.cc:407] Connection header received
0620 14:59:09.213846 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:09.213871 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:09.213978 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:09.214103 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:09.215677 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:09.216442 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:09.218209 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:09.219175 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:09.219304 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:09.223140 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:09.223147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:09.223152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:09.223193 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:09.228194 (+ 5001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:09.229409 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:09.229419 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:09.229421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:09.229561 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:09.230179 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:09.230186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:09.230194 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:09.230760 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:09.232530 (+ 1770us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:09.232738 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 14:59:09.232968 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":65}
I20260620 14:59:09.235495 11186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:09.244797 11186 catalog_manager.cc:1347] Generated new cluster ID: 4dc8602e0030410dab594e6798f344bc
I20260620 14:59:09.244962 11186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:09.256255 11186 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:09.258253 11186 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:09.265720 11186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef30cc846e244f6b8aedb902590d89be: Generated new TSK 0
I20260620 14:59:09.266839 11186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:09.276604 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11103
I20260620 14:59:09.278363 11165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:09.374408 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11103
W20260620 14:59:09.392181 11088 connection.cc:535] client connection to 127.0.0.1:39311 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39311): Connection reset by peer (error 104)
2026-06-20T14:59:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2092 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:09 dist-test-slave-3mqx krb5kdc[11202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:09 dist-test-slave-3mqx krb5kdc[11202](info): set up 2 sockets
Jun 20 14:59:09 dist-test-slave-3mqx krb5kdc[11202](info): commencing operation
krb5kdc: starting...
W20260620 14:59:10.563603 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:10 dist-test-slave-3mqx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:59:10.834141 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:11.177776 11215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:11.178180 11215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:11.178355 11215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:11.185988 11215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:11.186182 11215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:11.186300 11215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:11.186420 11215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:11.186501 11215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:11.197651 11215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:11.199270 11215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:11.200712 11215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:11.214227 11221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:11.214264 11223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:11.214298 11220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:11.215183 11215 server_base.cc:1034] running on GCE node
I20260620 14:59:11.216095 11215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:11.217844 11215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:11.219163 11215 hybrid_clock.cc:648] HybridClock initialized: now 1781967551219114 us; error 47 us; skew 500 ppm
I20260620 14:59:11.219587 11215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:11 dist-test-slave-3mqx krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:11.227535 11215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:59:11.229995 11215 webserver.cc:460] Webserver started at http://127.0.0.1:38685/ using document root <none> and password file <none>
I20260620 14:59:11.230928 11215 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:11.231182 11215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:11.231583 11215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:11.235464 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "a28a077bba6c40e4877908b76ed123ae"
format_stamp: "Formatted at 2026-06-20 14:59:11 on dist-test-slave-3mqx"
server_key: "b07614ab6b9557c2a8eff335be82f04f"
server_key_iv: "04a0aad54b1e8a1f2dad7c7185cb985b"
server_key_version: "encryptionkey@0"
I20260620 14:59:11.236727 11215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "a28a077bba6c40e4877908b76ed123ae"
format_stamp: "Formatted at 2026-06-20 14:59:11 on dist-test-slave-3mqx"
server_key: "b07614ab6b9557c2a8eff335be82f04f"
server_key_iv: "04a0aad54b1e8a1f2dad7c7185cb985b"
server_key_version: "encryptionkey@0"
I20260620 14:59:11.243510 11215 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 14:59:11.247877 11230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:11.249643 11215 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 14:59:11.249919 11215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "a28a077bba6c40e4877908b76ed123ae"
format_stamp: "Formatted at 2026-06-20 14:59:11 on dist-test-slave-3mqx"
server_key: "b07614ab6b9557c2a8eff335be82f04f"
server_key_iv: "04a0aad54b1e8a1f2dad7c7185cb985b"
server_key_version: "encryptionkey@0"
I20260620 14:59:11.250166 11215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:11.294281 11215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:11.299336 11215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:11.299664 11215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:11.320022 11215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40277
I20260620 14:59:11.320077 11281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40277 every 8 connection(s)
I20260620 14:59:11.321650 11215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:11.326170 11282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:11.332083 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11215
I20260620 14:59:11.332331 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:11.332923 22143 external_mini_cluster.cc:1428] Setting key 9a5c3e8141bf7de882c5d91f94a8da65
I20260620 14:59:11.342016 11282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae: Bootstrap starting.
I20260620 14:59:11.345898 11282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:11.347425 11282 log.cc:824] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:11.347810 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:11.335181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:59327 (local address 127.0.0.1:40277)
0620 14:59:11.336056 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:11.336069 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:11.336113 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:59:11.338036 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:11.338063 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:11.338568 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:11.339035 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:11.339691 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:11.341039 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:11.342469 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:11.343301 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:11.343538 (+ 237us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:11.345082 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:11.345091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:11.345109 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:11.345723 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:11.345818 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:11.345848 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:11.346053 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:11.346402 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 14:59:11.346594 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":359,"threads_started":1}
I20260620 14:59:11.350903 11282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae: No bootstrap required, opened a new log
I20260620 14:59:11.352130 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:11.361897 11282 raft_consensus.cc:348] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28a077bba6c40e4877908b76ed123ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40277 } }
I20260620 14:59:11.362498 11282 raft_consensus.cc:374] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:11.362632 11282 raft_consensus.cc:729] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28a077bba6c40e4877908b76ed123ae, State: Initialized, Role: FOLLOWER
I20260620 14:59:11.363530 11282 consensus_queue.cc:260] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28a077bba6c40e4877908b76ed123ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40277 } }
I20260620 14:59:11.363878 11282 raft_consensus.cc:388] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:11.364058 11282 raft_consensus.cc:482] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:11.364306 11282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:11.365231 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:11.356957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:39035 (local address 127.0.0.1:40277)
0620 14:59:11.357228 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:11.357235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:11.357716 (+ 481us) server_negotiation.cc:407] Connection header received
0620 14:59:11.357956 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:11.357960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:11.358040 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:11.358148 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:11.358866 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:11.359875 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:11.362056 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:11.362618 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:11.362775 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:11.364168 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:11.364174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:11.364177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:11.364199 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:11.364276 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:11.364305 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:11.364403 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:11.364733 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:59:11.364838 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:59:11.367717 11282 raft_consensus.cc:504] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28a077bba6c40e4877908b76ed123ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40277 } }
I20260620 14:59:11.368345 11282 leader_election.cc:302] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a28a077bba6c40e4877908b76ed123ae; no voters:
I20260620 14:59:11.369010 11282 leader_election.cc:288] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:11.369216 11292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:11.370139 11292 raft_consensus.cc:686] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [term 1 LEADER]: Becoming Leader. State: Replica: a28a077bba6c40e4877908b76ed123ae, State: Running, Role: LEADER
I20260620 14:59:11.370711 11292 consensus_queue.cc:237] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28a077bba6c40e4877908b76ed123ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40277 } }
I20260620 14:59:11.371704 11282 sys_catalog.cc:564] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:11.373688 11293 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a28a077bba6c40e4877908b76ed123ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28a077bba6c40e4877908b76ed123ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40277 } } }
I20260620 14:59:11.374156 11293 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:11.374774 11294 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader a28a077bba6c40e4877908b76ed123ae. Latest consensus state: current_term: 1 leader_uuid: "a28a077bba6c40e4877908b76ed123ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28a077bba6c40e4877908b76ed123ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40277 } } }
I20260620 14:59:11.375110 11294 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:11.375195 11297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:11.382931 11297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:11.393914 11297 catalog_manager.cc:1347] Generated new cluster ID: 789d2fe551584f65a3fd83deec9b6559
I20260620 14:59:11.394063 11297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:11.407572 11297 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:11.409019 11297 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:11.418279 11297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a28a077bba6c40e4877908b76ed123ae: Generated new TSK 0
I20260620 14:59:11.419576 11297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:11.426054 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11215
W20260620 14:59:11.427592 11277 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:59327
W20260620 14:59:11.428647 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11215: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:11.428843 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11215
W20260620 14:59:11.448266 11200 connection.cc:535] client connection to 127.0.0.1:40277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40277): Connection reset by peer (error 104)
2026-06-20T14:59:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:11 dist-test-slave-3mqx krb5kdc[11313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:11 dist-test-slave-3mqx krb5kdc[11313](info): set up 2 sockets
Jun 20 14:59:11 dist-test-slave-3mqx krb5kdc[11313](info): commencing operation
krb5kdc: starting...
W20260620 14:59:12.655704 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:12 dist-test-slave-3mqx krb5kdc[11313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:59:12.934795 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:13.303299 11326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:13.303620 11326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:13.303835 11326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:13.311566 11326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:13.311693 11326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:13.311802 11326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:13.311911 11326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:13.311995 11326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:13.322348 11326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:13.323902 11326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:13.325309 11326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:13.337903 11331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:13.337940 11332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:13.337908 11334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:13.339198 11326 server_base.cc:1034] running on GCE node
I20260620 14:59:13.340375 11326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:13.342466 11326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:13.343730 11326 hybrid_clock.cc:648] HybridClock initialized: now 1781967553343693 us; error 53 us; skew 500 ppm
I20260620 14:59:13.344130 11326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:13 dist-test-slave-3mqx krb5kdc[11313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:13.347543 11326 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:59:13.349136 11326 webserver.cc:460] Webserver started at http://127.0.0.1:33291/ using document root <none> and password file <none>
I20260620 14:59:13.349869 11326 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:13.350040 11326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:13.350373 11326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:13.353662 11326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "e4d9fd3b476a489fbef062554a487a85"
format_stamp: "Formatted at 2026-06-20 14:59:13 on dist-test-slave-3mqx"
server_key: "87033d8a7ef63995df509a62b4b1ff31"
server_key_iv: "b83c7f037446a544e59f9e58b21fb2db"
server_key_version: "encryptionkey@0"
I20260620 14:59:13.354544 11326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "e4d9fd3b476a489fbef062554a487a85"
format_stamp: "Formatted at 2026-06-20 14:59:13 on dist-test-slave-3mqx"
server_key: "87033d8a7ef63995df509a62b4b1ff31"
server_key_iv: "b83c7f037446a544e59f9e58b21fb2db"
server_key_version: "encryptionkey@0"
I20260620 14:59:13.360316 11326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:59:13.364511 11341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:13.366175 11326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:59:13.366436 11326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "e4d9fd3b476a489fbef062554a487a85"
format_stamp: "Formatted at 2026-06-20 14:59:13 on dist-test-slave-3mqx"
server_key: "87033d8a7ef63995df509a62b4b1ff31"
server_key_iv: "b83c7f037446a544e59f9e58b21fb2db"
server_key_version: "encryptionkey@0"
I20260620 14:59:13.366698 11326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:13.373409 11326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:13.378428 11326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:13.378757 11326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:13.398497 11326 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42113
I20260620 14:59:13.398543 11392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42113 every 8 connection(s)
I20260620 14:59:13.400221 11326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:13.402158 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11326
I20260620 14:59:13.402343 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:13.402884 22143 external_mini_cluster.cc:1428] Setting key ad2917a054dc13bff57ab0489e9bd51b
I20260620 14:59:13.405519 11393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:13.411520 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:13.404880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:55203 (local address 127.0.0.1:42113)
0620 14:59:13.405919 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:13.405932 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:13.405977 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:59:13.407734 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:13.407758 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:13.408132 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:13.408512 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:13.409713 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:13.409720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:13.409734 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:13.409752 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:13.409857 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:13.409882 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:13.410067 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:13.410180 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 14:59:13.410423 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":549,"threads_started":1}
I20260620 14:59:13.415748 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:13.418723 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85: Bootstrap starting.
I20260620 14:59:13.422490 11393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:13.423395 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:13.420563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:48949 (local address 127.0.0.1:42113)
0620 14:59:13.420800 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:13.420807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:13.421034 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:59:13.421277 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:13.421283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:13.421352 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:13.421473 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:13.422582 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:13.422588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:13.422594 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:13.422616 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:13.422677 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:13.422707 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:13.422842 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:13.422989 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:59:13.423092 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 14:59:13.423959 11393 log.cc:824] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:13.427239 11393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85: No bootstrap required, opened a new log
I20260620 14:59:13.435806 11393 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d9fd3b476a489fbef062554a487a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260620 14:59:13.436278 11393 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:13.436396 11393 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d9fd3b476a489fbef062554a487a85, State: Initialized, Role: FOLLOWER
I20260620 14:59:13.437083 11393 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d9fd3b476a489fbef062554a487a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260620 14:59:13.437345 11393 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:13.437497 11393 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:13.437688 11393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:13.440143 11393 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d9fd3b476a489fbef062554a487a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260620 14:59:13.440699 11393 leader_election.cc:302] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4d9fd3b476a489fbef062554a487a85; no voters:
I20260620 14:59:13.441453 11393 leader_election.cc:288] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:13.441615 11403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:13.442373 11403 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [term 1 LEADER]: Becoming Leader. State: Replica: e4d9fd3b476a489fbef062554a487a85, State: Running, Role: LEADER
I20260620 14:59:13.443176 11403 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d9fd3b476a489fbef062554a487a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } }
I20260620 14:59:13.444037 11393 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:13.446820 11404 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4d9fd3b476a489fbef062554a487a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d9fd3b476a489fbef062554a487a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } } }
I20260620 14:59:13.446949 11405 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4d9fd3b476a489fbef062554a487a85. Latest consensus state: current_term: 1 leader_uuid: "e4d9fd3b476a489fbef062554a487a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4d9fd3b476a489fbef062554a487a85" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42113 } } }
I20260620 14:59:13.447283 11404 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:13.447371 11405 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:13.448889 11409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:13.453902 11409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:13.464493 11409 catalog_manager.cc:1347] Generated new cluster ID: e5df2faf411845d495b38685be3e2e48
I20260620 14:59:13.464629 11409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:13.509132 11409 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:13.509635 11409 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:13.522738 11409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4d9fd3b476a489fbef062554a487a85: Generated new TSK 0
I20260620 14:59:13.524029 11409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:13.544044 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11326
W20260620 14:59:13.545378 11388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.164:55203
W20260620 14:59:13.546280 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11326: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:13.546471 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11326
2026-06-20T14:59:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2116 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:13 dist-test-slave-3mqx krb5kdc[11424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:13 dist-test-slave-3mqx krb5kdc[11424](info): set up 2 sockets
Jun 20 14:59:13 dist-test-slave-3mqx krb5kdc[11424](info): commencing operation
krb5kdc: starting...
W20260620 14:59:14.711735 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:14 dist-test-slave-3mqx krb5kdc[11424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:14.987828 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33579
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:15.338058 11437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:15.338385 11437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:15.338480 11437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:15.346077 11437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:15.346208 11437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:15.346256 11437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:15.346312 11437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:15.346364 11437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:15.358072 11437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33579
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:15.359510 11437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:15.360865 11437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:15.373574 11442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:15.373654 11443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:15.373714 11445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:15.374883 11437 server_base.cc:1034] running on GCE node
I20260620 14:59:15.375810 11437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:15.377369 11437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:15.378618 11437 hybrid_clock.cc:648] HybridClock initialized: now 1781967555378573 us; error 48 us; skew 500 ppm
I20260620 14:59:15.379019 11437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:15 dist-test-slave-3mqx krb5kdc[11424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967555, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:15.382939 11437 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:15.384742 11437 webserver.cc:460] Webserver started at http://127.21.159.254:36721/ using document root <none> and password file <none>
I20260620 14:59:15.385511 11437 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:15.385650 11437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:15.385984 11437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:15.389222 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7"
format_stamp: "Formatted at 2026-06-20 14:59:15 on dist-test-slave-3mqx"
server_key: "2dec11a3833b58b22e3d663844f6eaa5"
server_key_iv: "81d885ffab6de839eb3ec1d0c2aecfac"
server_key_version: "encryptionkey@0"
I20260620 14:59:15.390115 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7"
format_stamp: "Formatted at 2026-06-20 14:59:15 on dist-test-slave-3mqx"
server_key: "2dec11a3833b58b22e3d663844f6eaa5"
server_key_iv: "81d885ffab6de839eb3ec1d0c2aecfac"
server_key_version: "encryptionkey@0"
I20260620 14:59:15.395958 11437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:59:15.399921 11452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:15.401443 11437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:59:15.401643 11437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7"
format_stamp: "Formatted at 2026-06-20 14:59:15 on dist-test-slave-3mqx"
server_key: "2dec11a3833b58b22e3d663844f6eaa5"
server_key_iv: "81d885ffab6de839eb3ec1d0c2aecfac"
server_key_version: "encryptionkey@0"
I20260620 14:59:15.401845 11437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:15.422770 11437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:15.427351 11437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:15.427672 11437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:15.447631 11437 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:33579
I20260620 14:59:15.447695 11503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:33579 every 8 connection(s)
I20260620 14:59:15.449201 11437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:15.453603 11504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:15.454952 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11437
I20260620 14:59:15.455197 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:15.455861 22143 external_mini_cluster.cc:1428] Setting key 07c63b89a911729804174c126edcc08f
I20260620 14:59:15.470160 11504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7: Bootstrap starting.
Jun 20 14:59:15 dist-test-slave-3mqx krb5kdc[11424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:59:15.474431 11504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:15.475962 11504 log.cc:824] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:15.479396 11504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7: No bootstrap required, opened a new log
I20260620 14:59:15.481999 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:15.457936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35185 (local address 127.21.159.254:33579)
0620 14:59:15.458790 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:15.458807 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:15.458854 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:59:15.460809 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:15.460840 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:15.461336 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:15.461722 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:15.463969 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:15.465388 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:15.467489 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:15.468239 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:15.468381 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:15.473790 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:15.473799 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:15.473817 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:15.473863 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:15.477328 (+ 3465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:15.478250 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:15.478257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:15.478265 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:15.478455 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:15.479002 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:15.479006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:15.479008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:15.479665 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:15.480039 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:15.480568 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 14:59:15.480912 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":404,"threads_started":1}
I20260620 14:59:15.486356 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:15.489125 11504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33579 } }
I20260620 14:59:15.489692 11504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:15.489845 11504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5b6fe757ae43f5ad5dbcdc47ca4cc7, State: Initialized, Role: FOLLOWER
I20260620 14:59:15.490581 11504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33579 } }
I20260620 14:59:15.490942 11504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:15.491163 11504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:15.491415 11504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:15.494233 11504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33579 } }
I20260620 14:59:15.494958 11504 leader_election.cc:302] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5b6fe757ae43f5ad5dbcdc47ca4cc7; no voters:
I20260620 14:59:15.495814 11504 leader_election.cc:288] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:15.495942 11514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:15.498138 11514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [term 1 LEADER]: Becoming Leader. State: Replica: 3a5b6fe757ae43f5ad5dbcdc47ca4cc7, State: Running, Role: LEADER
I20260620 14:59:15.498787 11504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:15.498868 11514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33579 } }
I20260620 14:59:15.502638 11516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a5b6fe757ae43f5ad5dbcdc47ca4cc7. Latest consensus state: current_term: 1 leader_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33579 } } }
I20260620 14:59:15.503237 11516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:15.507771 11515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5b6fe757ae43f5ad5dbcdc47ca4cc7" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33579 } } }
I20260620 14:59:15.509068 11515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:15.510694 11507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:15.491126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41737 (local address 127.21.159.254:33579)
0620 14:59:15.491333 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:15.491339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:15.491858 (+ 519us) server_negotiation.cc:407] Connection header received
0620 14:59:15.492078 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:15.492083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:15.492168 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:15.492288 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:15.493831 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:15.494708 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:15.496340 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:15.497052 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:15.497159 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:15.500262 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:15.500269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:15.500274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:15.500313 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:15.504545 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:15.506249 (+ 1704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:15.506258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:15.506265 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:15.506471 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:15.507333 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:15.507341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:15.507346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:15.507903 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:15.509595 (+ 1692us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:15.509846 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 14:59:15.510122 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:59:15.514534 11519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:15.520671 11519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:15.532167 11519 catalog_manager.cc:1347] Generated new cluster ID: b4c1f96f7614408ab6281844e710fbad
I20260620 14:59:15.532321 11519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:15.543776 11519 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:15.545668 11519 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:15.561523 11519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a5b6fe757ae43f5ad5dbcdc47ca4cc7: Generated new TSK 0
I20260620 14:59:15.562602 11519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:15.568678 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11437
I20260620 14:59:15.570442 11499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:15.664438 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11437
W20260620 14:59:15.682279 11422 connection.cc:535] client connection to 127.21.159.254:33579 recv error: Network error: failed to read from TLS socket (remote: 127.21.159.254:33579): Connection reset by peer (error 104)
2026-06-20T14:59:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2117 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:15 dist-test-slave-3mqx krb5kdc[11536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:15 dist-test-slave-3mqx krb5kdc[11536](info): set up 2 sockets
Jun 20 14:59:15 dist-test-slave-3mqx krb5kdc[11536](info): commencing operation
krb5kdc: starting...
W20260620 14:59:16.791652 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:16 dist-test-slave-3mqx krb5kdc[11536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:17.067137 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33079
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:17.420222 11549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:17.420600 11549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:17.420761 11549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:17.428223 11549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:17.428401 11549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:17.428519 11549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:17.428625 11549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:17.428710 11549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:17.439113 11549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:17.440590 11549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:17.441996 11549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:17.454798 11555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:17.454766 11554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:17.454859 11557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:17.455722 11549 server_base.cc:1034] running on GCE node
I20260620 14:59:17.456575 11549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:17.458200 11549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:17.459451 11549 hybrid_clock.cc:648] HybridClock initialized: now 1781967557459409 us; error 37 us; skew 500 ppm
I20260620 14:59:17.459815 11549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:17 dist-test-slave-3mqx krb5kdc[11536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967557, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:17.467756 11549 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:17.469568 11549 webserver.cc:460] Webserver started at http://127.21.159.254:43951/ using document root <none> and password file <none>
I20260620 14:59:17.470350 11549 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:17.470515 11549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:17.470854 11549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:17.474213 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "278eb24f8479499ebb2a1bc26ddeffe8"
format_stamp: "Formatted at 2026-06-20 14:59:17 on dist-test-slave-3mqx"
server_key: "145fb748ca9b6ff4884522eddfbb6fc0"
server_key_iv: "92cace485d3128110de1db1b5d9b3625"
server_key_version: "encryptionkey@0"
I20260620 14:59:17.475009 11549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "278eb24f8479499ebb2a1bc26ddeffe8"
format_stamp: "Formatted at 2026-06-20 14:59:17 on dist-test-slave-3mqx"
server_key: "145fb748ca9b6ff4884522eddfbb6fc0"
server_key_iv: "92cace485d3128110de1db1b5d9b3625"
server_key_version: "encryptionkey@0"
I20260620 14:59:17.480792 11549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:59:17.484727 11564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:17.486330 11549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:59:17.486567 11549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "278eb24f8479499ebb2a1bc26ddeffe8"
format_stamp: "Formatted at 2026-06-20 14:59:17 on dist-test-slave-3mqx"
server_key: "145fb748ca9b6ff4884522eddfbb6fc0"
server_key_iv: "92cace485d3128110de1db1b5d9b3625"
server_key_version: "encryptionkey@0"
I20260620 14:59:17.486805 11549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:17.509101 11549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:17.514137 11549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:17.514441 11549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:17.535794 11549 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:33079
I20260620 14:59:17.535842 11615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:33079 every 8 connection(s)
I20260620 14:59:17.537284 11549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:17.541298 11616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:17.544873 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11549
I20260620 14:59:17.545037 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:17.545568 22143 external_mini_cluster.cc:1428] Setting key 3e759d62e0b145dea26f08c7f59145ea
I20260620 14:59:17.555763 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8: Bootstrap starting.
I20260620 14:59:17.559942 11616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:17.561429 11616 log.cc:824] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:17.561774 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:17.547708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59937 (local address 127.21.159.254:33079)
0620 14:59:17.548525 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:17.548541 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:17.548594 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:59:17.550929 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:17.550968 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:17.551619 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:17.552125 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:17.552691 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:17.554747 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:17.556359 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:17.557385 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:17.557604 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:17.558873 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:17.558884 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:17.558902 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:17.559695 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:17.559800 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:17.559852 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:17.560033 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:17.560448 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 14:59:17.560627 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":333,"threads_started":1}
I20260620 14:59:17.564894 11616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8: No bootstrap required, opened a new log
I20260620 14:59:17.566154 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:17.573676 11616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33079 } }
I20260620 14:59:17.574108 11616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:17.574213 11616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278eb24f8479499ebb2a1bc26ddeffe8, State: Initialized, Role: FOLLOWER
I20260620 14:59:17.574918 11616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33079 } }
I20260620 14:59:17.575280 11616 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:17.575438 11616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:17.575636 11616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:17.577980 11619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:17.570989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53235 (local address 127.21.159.254:33079)
0620 14:59:17.571231 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:17.571241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:17.571502 (+ 261us) server_negotiation.cc:407] Connection header received
0620 14:59:17.571696 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:17.571702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:17.571790 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:17.571923 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:17.572433 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:17.573516 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:17.575020 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:17.575744 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:17.575908 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:17.577123 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:17.577129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:17.577134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:17.577168 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:17.577239 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:17.577266 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:17.577375 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:17.577587 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:59:17.577650 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 14:59:17.578600 11616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33079 } }
I20260620 14:59:17.579308 11616 leader_election.cc:302] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278eb24f8479499ebb2a1bc26ddeffe8; no voters:
I20260620 14:59:17.580063 11616 leader_election.cc:288] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:17.580168 11626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:17.580902 11626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [term 1 LEADER]: Becoming Leader. State: Replica: 278eb24f8479499ebb2a1bc26ddeffe8, State: Running, Role: LEADER
I20260620 14:59:17.581560 11626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33079 } }
I20260620 14:59:17.582633 11616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:17.584856 11628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278eb24f8479499ebb2a1bc26ddeffe8. Latest consensus state: current_term: 1 leader_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33079 } } }
I20260620 14:59:17.584949 11627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278eb24f8479499ebb2a1bc26ddeffe8" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33079 } } }
I20260620 14:59:17.585400 11628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:17.585413 11627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:17.589483 11631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:17.595642 11631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:17.606890 11631 catalog_manager.cc:1347] Generated new cluster ID: 7e77d40b93be40b3852d7873c20dd17b
I20260620 14:59:17.607043 11631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:17.622720 11631 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:17.624709 11631 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:17.638164 11631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278eb24f8479499ebb2a1bc26ddeffe8: Generated new TSK 0
I20260620 14:59:17.639513 11631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:17.649551 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11549
W20260620 14:59:17.650964 11611 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59937
W20260620 14:59:17.651872 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:17.652053 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11549
W20260620 14:59:17.669371 11534 connection.cc:535] client connection to 127.21.159.254:33079 recv error: Network error: failed to read from TLS socket (remote: 127.21.159.254:33079): Connection reset by peer (error 104)
2026-06-20T14:59:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:17 dist-test-slave-3mqx krb5kdc[11647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:17 dist-test-slave-3mqx krb5kdc[11647](info): set up 2 sockets
Jun 20 14:59:17 dist-test-slave-3mqx krb5kdc[11647](info): commencing operation
krb5kdc: starting...
W20260620 14:59:18.856171 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:19 dist-test-slave-3mqx krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:19.135222 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38701
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:38701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:19.495136 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:19.495462 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:19.495652 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:19.503275 11660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:19.503400 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:19.503475 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:19.503574 11660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:19.503675 11660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:19.514195 11660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:38701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:38701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:19.515648 11660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:19.517033 11660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:19.530040 11665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:19.530021 11666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:19.530124 11668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:19.530877 11660 server_base.cc:1034] running on GCE node
I20260620 14:59:19.531894 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:19.533561 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:19.534837 11660 hybrid_clock.cc:648] HybridClock initialized: now 1781967559534777 us; error 68 us; skew 500 ppm
I20260620 14:59:19.535403 11660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:19 dist-test-slave-3mqx krb5kdc[11647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967559, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:19.538995 11660 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:19.540764 11660 webserver.cc:460] Webserver started at http://127.21.159.254:46159/ using document root <none> and password file <none>
I20260620 14:59:19.541510 11660 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:19.541621 11660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:19.541891 11660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:19.545168 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "dd6aa771ffab41f8b2de72713d86d208"
format_stamp: "Formatted at 2026-06-20 14:59:19 on dist-test-slave-3mqx"
server_key: "f51867c47b6aa1430e00eca45bac166b"
server_key_iv: "648a98b0a62545e82c46ed3e2938d12b"
server_key_version: "encryptionkey@0"
I20260620 14:59:19.545958 11660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "dd6aa771ffab41f8b2de72713d86d208"
format_stamp: "Formatted at 2026-06-20 14:59:19 on dist-test-slave-3mqx"
server_key: "f51867c47b6aa1430e00eca45bac166b"
server_key_iv: "648a98b0a62545e82c46ed3e2938d12b"
server_key_version: "encryptionkey@0"
I20260620 14:59:19.551766 11660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:59:19.555879 11675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:19.557636 11660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:59:19.557853 11660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "dd6aa771ffab41f8b2de72713d86d208"
format_stamp: "Formatted at 2026-06-20 14:59:19 on dist-test-slave-3mqx"
server_key: "f51867c47b6aa1430e00eca45bac166b"
server_key_iv: "648a98b0a62545e82c46ed3e2938d12b"
server_key_version: "encryptionkey@0"
I20260620 14:59:19.558086 11660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:19.565052 11660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:19.570106 11660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:19.570399 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:19.589918 11660 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:38701
I20260620 14:59:19.589948 11726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:38701 every 8 connection(s)
I20260620 14:59:19.591535 11660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:19.592196 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11660
I20260620 14:59:19.592394 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:19.593035 22143 external_mini_cluster.cc:1428] Setting key df324dee51408b69242ac68e71863c41
I20260620 14:59:19.596220 11727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:19.601227 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:19.594952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46387 (local address 127.21.159.254:38701)
0620 14:59:19.595842 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:19.595855 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:19.595897 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:59:19.597614 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:19.597638 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:19.597984 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:19.598326 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:19.599565 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:19.599572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:19.599585 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:19.599600 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:19.599680 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:19.599703 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:19.599883 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:19.599984 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:59:19.600194 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":374,"threads_started":1}
I20260620 14:59:19.605603 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:19.609244 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208: Bootstrap starting.
I20260620 14:59:19.612764 11727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:19.613070 11730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:19.610246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35475 (local address 127.21.159.254:38701)
0620 14:59:19.610496 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:19.610506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:19.611044 (+ 538us) server_negotiation.cc:407] Connection header received
0620 14:59:19.611297 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:19.611301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:19.611376 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:19.611489 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:19.612427 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:19.612433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:19.612437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:19.612455 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:19.612504 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:19.612519 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:19.612634 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:19.612730 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:59:19.612797 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:59:19.614179 11727 log.cc:824] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:19.617300 11727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208: No bootstrap required, opened a new log
I20260620 14:59:19.624984 11727 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6aa771ffab41f8b2de72713d86d208" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38701 } }
I20260620 14:59:19.625502 11727 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:19.625643 11727 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6aa771ffab41f8b2de72713d86d208, State: Initialized, Role: FOLLOWER
I20260620 14:59:19.626272 11727 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6aa771ffab41f8b2de72713d86d208" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38701 } }
I20260620 14:59:19.626526 11727 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:19.626677 11727 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:19.626852 11727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:19.629177 11727 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6aa771ffab41f8b2de72713d86d208" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38701 } }
I20260620 14:59:19.629711 11727 leader_election.cc:302] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6aa771ffab41f8b2de72713d86d208; no voters:
I20260620 14:59:19.630375 11727 leader_election.cc:288] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:19.630621 11737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:19.631381 11737 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [term 1 LEADER]: Becoming Leader. State: Replica: dd6aa771ffab41f8b2de72713d86d208, State: Running, Role: LEADER
I20260620 14:59:19.632043 11737 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6aa771ffab41f8b2de72713d86d208" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38701 } }
I20260620 14:59:19.632815 11727 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:19.635115 11738 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6aa771ffab41f8b2de72713d86d208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6aa771ffab41f8b2de72713d86d208" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38701 } } }
I20260620 14:59:19.635349 11739 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6aa771ffab41f8b2de72713d86d208. Latest consensus state: current_term: 1 leader_uuid: "dd6aa771ffab41f8b2de72713d86d208" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6aa771ffab41f8b2de72713d86d208" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 38701 } } }
I20260620 14:59:19.635605 11739 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:19.635607 11738 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:19.637686 11745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:19.643004 11745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:19.653823 11745 catalog_manager.cc:1347] Generated new cluster ID: 4711872873024bbca1f75649b8ba35ff
I20260620 14:59:19.653966 11745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:19.670985 11745 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:19.671592 11745 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:19.688657 11745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd6aa771ffab41f8b2de72713d86d208: Generated new TSK 0
I20260620 14:59:19.689637 11745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:19.704535 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11660
W20260620 14:59:19.706048 11722 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46387
W20260620 14:59:19.707147 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11660: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:19.707347 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11660
2026-06-20T14:59:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:19 dist-test-slave-3mqx krb5kdc[11758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:19 dist-test-slave-3mqx krb5kdc[11758](info): set up 2 sockets
Jun 20 14:59:19 dist-test-slave-3mqx krb5kdc[11758](info): commencing operation
krb5kdc: starting...
W20260620 14:59:20.860038 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:21 dist-test-slave-3mqx krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:21.130525 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:41837
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:46821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:41837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:21.482002 11771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:21.482399 11771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:21.482580 11771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:21.490311 11771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:21.490499 11771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:21.490622 11771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:21.490734 11771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:21.490814 11771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:21.501597 11771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:46821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:41837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:41837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:21.503206 11771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:21.504644 11771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:21.517916 11777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:21.518177 11776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:21.517966 11779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:21.518946 11771 server_base.cc:1034] running on GCE node
I20260620 14:59:21.519889 11771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:21.521570 11771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:21.522864 11771 hybrid_clock.cc:648] HybridClock initialized: now 1781967561522805 us; error 51 us; skew 500 ppm
I20260620 14:59:21.523367 11771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:21 dist-test-slave-3mqx krb5kdc[11758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967561, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:21.527217 11771 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:21.529011 11771 webserver.cc:460] Webserver started at http://127.21.159.254:40469/ using document root <none> and password file <none>
I20260620 14:59:21.529838 11771 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:21.530010 11771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:21.530360 11771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:21.534049 11771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "10df6814ebfe4c949034a07375ed1b4c"
format_stamp: "Formatted at 2026-06-20 14:59:21 on dist-test-slave-3mqx"
server_key: "96fec49773d80a383d290f8c711940a6"
server_key_iv: "5484d56252ed8217ea96a36a657d5871"
server_key_version: "encryptionkey@0"
I20260620 14:59:21.535107 11771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "10df6814ebfe4c949034a07375ed1b4c"
format_stamp: "Formatted at 2026-06-20 14:59:21 on dist-test-slave-3mqx"
server_key: "96fec49773d80a383d290f8c711940a6"
server_key_iv: "5484d56252ed8217ea96a36a657d5871"
server_key_version: "encryptionkey@0"
I20260620 14:59:21.541661 11771 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:59:21.546170 11786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:21.547864 11771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:59:21.548144 11771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "10df6814ebfe4c949034a07375ed1b4c"
format_stamp: "Formatted at 2026-06-20 14:59:21 on dist-test-slave-3mqx"
server_key: "96fec49773d80a383d290f8c711940a6"
server_key_iv: "5484d56252ed8217ea96a36a657d5871"
server_key_version: "encryptionkey@0"
I20260620 14:59:21.548391 11771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:21.589008 11771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:21.594262 11771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:21.594650 11771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:21.616879 11771 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:41837
I20260620 14:59:21.616905 11837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:41837 every 8 connection(s)
I20260620 14:59:21.618531 11771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:21.619232 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11771
I20260620 14:59:21.619443 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:21.620143 22143 external_mini_cluster.cc:1428] Setting key bcd4eebd59f22012170325a65b336a8c
I20260620 14:59:21.623857 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:59:21 dist-test-slave-3mqx krb5kdc[11758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:59:21.636605 11838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c: Bootstrap starting.
I20260620 14:59:21.641714 11838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:21.643313 11838 log.cc:824] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:21.647473 11838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c: No bootstrap required, opened a new log
I20260620 14:59:21.648043 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:21.622221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49217 (local address 127.21.159.254:41837)
0620 14:59:21.623386 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:21.623405 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:21.623454 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:59:21.625451 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:21.625480 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:21.625948 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:21.626423 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:21.628221 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:21.629842 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:21.631553 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:21.632377 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:21.632618 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:21.637499 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:21.637514 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:21.637537 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:21.637602 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:21.642477 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:21.643541 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:21.643553 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:21.643564 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:21.643821 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:21.644359 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:21.644363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:21.644366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:21.645170 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:21.645501 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:21.645939 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 14:59:21.646470 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":443,"threads_started":1}
I20260620 14:59:21.652812 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:21.658454 11838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df6814ebfe4c949034a07375ed1b4c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41837 } }
I20260620 14:59:21.658946 11838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:21.659178 11838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10df6814ebfe4c949034a07375ed1b4c, State: Initialized, Role: FOLLOWER
I20260620 14:59:21.660000 11838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df6814ebfe4c949034a07375ed1b4c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41837 } }
I20260620 14:59:21.660305 11838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:21.660493 11838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:21.660715 11838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:21.663422 11838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df6814ebfe4c949034a07375ed1b4c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41837 } }
I20260620 14:59:21.664120 11838 leader_election.cc:302] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10df6814ebfe4c949034a07375ed1b4c; no voters:
I20260620 14:59:21.664887 11838 leader_election.cc:288] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:21.665004 11848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:21.667341 11848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [term 1 LEADER]: Becoming Leader. State: Replica: 10df6814ebfe4c949034a07375ed1b4c, State: Running, Role: LEADER
I20260620 14:59:21.667953 11838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:21.668128 11848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df6814ebfe4c949034a07375ed1b4c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41837 } }
I20260620 14:59:21.671711 11850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10df6814ebfe4c949034a07375ed1b4c. Latest consensus state: current_term: 1 leader_uuid: "10df6814ebfe4c949034a07375ed1b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df6814ebfe4c949034a07375ed1b4c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41837 } } }
I20260620 14:59:21.672228 11850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:21.672026 11849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10df6814ebfe4c949034a07375ed1b4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10df6814ebfe4c949034a07375ed1b4c" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 41837 } } }
I20260620 14:59:21.672451 11849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:21.676630 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:21.657788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59439 (local address 127.21.159.254:41837)
0620 14:59:21.658030 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:21.658039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:21.658511 (+ 472us) server_negotiation.cc:407] Connection header received
0620 14:59:21.658782 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:21.658787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:21.658904 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:21.659108 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:21.660977 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:21.661849 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:21.663414 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:21.663968 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:21.664143 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:21.667384 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:21.667394 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:21.667405 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:21.667461 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:21.671591 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:21.672680 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:21.672693 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:21.672697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:21.673012 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:21.673736 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:21.673746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:21.673752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:21.674334 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:21.674731 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:21.675641 (+ 910us) server_negotiation.cc:299] Negotiation successful
0620 14:59:21.676006 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:59:21.681844 11857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:21.687749 11857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:21.697412 11857 catalog_manager.cc:1347] Generated new cluster ID: fc4f02c2f8f24f028cb93925001a3ce0
I20260620 14:59:21.697607 11857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:21.720616 11857 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:21.722038 11857 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:21.735450 11857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10df6814ebfe4c949034a07375ed1b4c: Generated new TSK 0
I20260620 14:59:21.736728 11857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:21.751017 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11771
I20260620 14:59:21.752380 11833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:21.849092 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11771
2026-06-20T14:59:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2140 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:21 dist-test-slave-3mqx krb5kdc[11870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:21 dist-test-slave-3mqx krb5kdc[11870](info): set up 2 sockets
Jun 20 14:59:21 dist-test-slave-3mqx krb5kdc[11870](info): commencing operation
krb5kdc: starting...
W20260620 14:59:23.023790 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:23 dist-test-slave-3mqx krb5kdc[11870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:23.318329 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:45399
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:35805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:45399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:23.667861 11883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:23.668190 11883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:23.668322 11883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:23.675699 11883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:23.675822 11883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:23.675868 11883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:23.675930 11883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:23.675976 11883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:23.686184 11883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:35805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:45399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:45399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:23.687624 11883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:23.689000 11883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:23.701923 11889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:23.702062 11891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:23.701885 11888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:23.702778 11883 server_base.cc:1034] running on GCE node
I20260620 14:59:23.703773 11883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:23.705487 11883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:23.706733 11883 hybrid_clock.cc:648] HybridClock initialized: now 1781967563706694 us; error 37 us; skew 500 ppm
I20260620 14:59:23.707165 11883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:23 dist-test-slave-3mqx krb5kdc[11870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967563, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:23.710526 11883 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:23.712327 11883 webserver.cc:460] Webserver started at http://127.21.159.254:44705/ using document root <none> and password file <none>
I20260620 14:59:23.713095 11883 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:23.713254 11883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:23.713593 11883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:23.716889 11883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "947b49c6812840e4b8f649bf10cb2f84"
format_stamp: "Formatted at 2026-06-20 14:59:23 on dist-test-slave-3mqx"
server_key: "cbfacbf359d329a21d71f979e4b736cb"
server_key_iv: "2adc5c90d8dde02f5bc03c0f85cfbb31"
server_key_version: "encryptionkey@0"
I20260620 14:59:23.717734 11883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "947b49c6812840e4b8f649bf10cb2f84"
format_stamp: "Formatted at 2026-06-20 14:59:23 on dist-test-slave-3mqx"
server_key: "cbfacbf359d329a21d71f979e4b736cb"
server_key_iv: "2adc5c90d8dde02f5bc03c0f85cfbb31"
server_key_version: "encryptionkey@0"
I20260620 14:59:23.723542 11883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 14:59:23.727613 11898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:23.729244 11883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:59:23.729465 11883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "947b49c6812840e4b8f649bf10cb2f84"
format_stamp: "Formatted at 2026-06-20 14:59:23 on dist-test-slave-3mqx"
server_key: "cbfacbf359d329a21d71f979e4b736cb"
server_key_iv: "2adc5c90d8dde02f5bc03c0f85cfbb31"
server_key_version: "encryptionkey@0"
I20260620 14:59:23.729714 11883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:23.750231 11883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:23.755311 11883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:23.755640 11883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:23.776695 11883 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:45399
I20260620 14:59:23.776701 11949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:45399 every 8 connection(s)
I20260620 14:59:23.778223 11883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:23.782249 11950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:23.786911 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11883
I20260620 14:59:23.787138 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:23.787585 22143 external_mini_cluster.cc:1428] Setting key e1d0e1d973f90388375bd353ce9d1ce1
I20260620 14:59:23.794232 11950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84: Bootstrap starting.
I20260620 14:59:23.798177 11950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:23.799561 11950 log.cc:824] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:23.802987 11950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84: No bootstrap required, opened a new log
I20260620 14:59:23.803248 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:23.789610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32785 (local address 127.21.159.254:45399)
0620 14:59:23.790479 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:23.790496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:23.790541 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:59:23.792950 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:23.792980 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:23.793490 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:23.793892 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:23.794595 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:23.796201 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:23.798111 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:23.798951 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:23.799192 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:23.800575 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:23.800588 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:23.800613 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:23.801373 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:23.801452 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:23.801507 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:23.801604 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:23.801856 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:59:23.802171 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":398,"threads_started":1}
I20260620 14:59:23.807291 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:23.811712 11950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b49c6812840e4b8f649bf10cb2f84" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45399 } }
I20260620 14:59:23.812263 11950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:23.812389 11950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 947b49c6812840e4b8f649bf10cb2f84, State: Initialized, Role: FOLLOWER
I20260620 14:59:23.813086 11950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b49c6812840e4b8f649bf10cb2f84" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45399 } }
I20260620 14:59:23.813346 11950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:23.813519 11950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:23.813712 11950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:23.816174 11950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b49c6812840e4b8f649bf10cb2f84" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45399 } }
I20260620 14:59:23.816726 11950 leader_election.cc:302] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 947b49c6812840e4b8f649bf10cb2f84; no voters:
I20260620 14:59:23.817540 11950 leader_election.cc:288] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:23.817652 11960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:23.818306 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:23.811651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51955 (local address 127.21.159.254:45399)
0620 14:59:23.811899 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:23.811905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:23.812479 (+ 574us) server_negotiation.cc:407] Connection header received
0620 14:59:23.812564 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:23.812570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:23.812647 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:23.812750 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:23.813374 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:23.814287 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:23.815721 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:23.816197 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:23.816340 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:23.817452 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:23.817457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:23.817461 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:23.817489 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:23.817530 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:23.817549 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:23.817688 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:23.817823 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:59:23.817918 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:59:23.819353 11960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [term 1 LEADER]: Becoming Leader. State: Replica: 947b49c6812840e4b8f649bf10cb2f84, State: Running, Role: LEADER
I20260620 14:59:23.820142 11960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b49c6812840e4b8f649bf10cb2f84" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45399 } }
I20260620 14:59:23.820459 11950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:23.823706 11961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "947b49c6812840e4b8f649bf10cb2f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b49c6812840e4b8f649bf10cb2f84" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45399 } } }
I20260620 14:59:23.823743 11962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 947b49c6812840e4b8f649bf10cb2f84. Latest consensus state: current_term: 1 leader_uuid: "947b49c6812840e4b8f649bf10cb2f84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "947b49c6812840e4b8f649bf10cb2f84" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 45399 } } }
I20260620 14:59:23.824147 11961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:23.824162 11962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:23.825054 11968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:23.833570 11968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:23.844517 11968 catalog_manager.cc:1347] Generated new cluster ID: 40a82cd861664438832b215d42f0b340
I20260620 14:59:23.844651 11968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:23.859076 11968 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:23.860486 11968 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:23.873608 11968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 947b49c6812840e4b8f649bf10cb2f84: Generated new TSK 0
I20260620 14:59:23.874650 11968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:23.889118 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11883
W20260620 14:59:23.890440 11945 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32785
W20260620 14:59:23.891366 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11883: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:23.891571 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11883
2026-06-20T14:59:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:23 dist-test-slave-3mqx krb5kdc[11981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:23 dist-test-slave-3mqx krb5kdc[11981](info): set up 2 sockets
Jun 20 14:59:23 dist-test-slave-3mqx krb5kdc[11981](info): commencing operation
krb5kdc: starting...
W20260620 14:59:25.055683 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:25 dist-test-slave-3mqx krb5kdc[11981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:25.338073 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:34969
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:34969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:25.692205 11994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:25.692569 11994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:25.692760 11994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:25.700510 11994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:25.700654 11994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:25.700789 11994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:25.700889 11994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:25.700978 11994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:25.711880 11994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:34969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:34969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:25.713364 11994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:25.714823 11994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:25.728636 12000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:25.728721 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:25.728718 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:25.729853 11994 server_base.cc:1034] running on GCE node
I20260620 14:59:25.730734 11994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:25.732373 11994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:25.733604 11994 hybrid_clock.cc:648] HybridClock initialized: now 1781967565733558 us; error 32 us; skew 500 ppm
I20260620 14:59:25.734001 11994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:25 dist-test-slave-3mqx krb5kdc[11981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967565, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:25.737392 11994 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:25.739225 11994 webserver.cc:460] Webserver started at http://127.21.159.254:43083/ using document root <none> and password file <none>
I20260620 14:59:25.739964 11994 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:25.740126 11994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:25.740451 11994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:25.743738 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "cb977b6a846c4ed49d7dec39eb5c86a1"
format_stamp: "Formatted at 2026-06-20 14:59:25 on dist-test-slave-3mqx"
server_key: "e70b774c47bf8f552bb7333dd95569c0"
server_key_iv: "629d8f0de94b7c7262a4a725d354738e"
server_key_version: "encryptionkey@0"
I20260620 14:59:25.744572 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "cb977b6a846c4ed49d7dec39eb5c86a1"
format_stamp: "Formatted at 2026-06-20 14:59:25 on dist-test-slave-3mqx"
server_key: "e70b774c47bf8f552bb7333dd95569c0"
server_key_iv: "629d8f0de94b7c7262a4a725d354738e"
server_key_version: "encryptionkey@0"
I20260620 14:59:25.750304 11994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 14:59:25.754323 12009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:25.755935 11994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:59:25.756168 11994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "cb977b6a846c4ed49d7dec39eb5c86a1"
format_stamp: "Formatted at 2026-06-20 14:59:25 on dist-test-slave-3mqx"
server_key: "e70b774c47bf8f552bb7333dd95569c0"
server_key_iv: "629d8f0de94b7c7262a4a725d354738e"
server_key_version: "encryptionkey@0"
I20260620 14:59:25.756394 11994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:25.762503 11994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:25.767274 11994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:25.767621 11994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:25.787701 11994 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:34969
I20260620 14:59:25.787725 12060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:34969 every 8 connection(s)
I20260620 14:59:25.789294 11994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:25.793499 12061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:25.795794 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 11994
I20260620 14:59:25.796000 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:25.796522 22143 external_mini_cluster.cc:1428] Setting key cd215d666d95a57f019d1917f37f43ea
I20260620 14:59:25.805661 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:25.798727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38309 (local address 127.21.159.254:34969)
0620 14:59:25.799752 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:25.799768 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:25.799814 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:59:25.801678 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:25.801704 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:25.802102 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:25.802575 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:25.803816 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:25.803825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:25.803844 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:25.803858 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:25.803921 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:25.803944 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:25.804230 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:25.804409 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:59:25.804637 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":427,"threads_started":1}
I20260620 14:59:25.806926 12061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1: Bootstrap starting.
I20260620 14:59:25.810017 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:25.810657 12061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:25.811831 12061 log.cc:824] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:25.815598 12061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1: No bootstrap required, opened a new log
I20260620 14:59:25.817631 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:25.814612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53503 (local address 127.21.159.254:34969)
0620 14:59:25.814773 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:25.814777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:25.815383 (+ 606us) server_negotiation.cc:407] Connection header received
0620 14:59:25.815604 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:25.815608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:25.815658 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:25.815766 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:25.816875 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:25.816880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:25.816884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:59:25.816916 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:25.816959 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:59:25.816975 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:25.817109 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:25.817244 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:59:25.817317 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:59:25.824361 12061 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34969 } }
I20260620 14:59:25.824831 12061 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:25.824945 12061 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb977b6a846c4ed49d7dec39eb5c86a1, State: Initialized, Role: FOLLOWER
I20260620 14:59:25.825645 12061 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34969 } }
I20260620 14:59:25.825870 12061 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:25.825964 12061 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:25.826123 12061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:25.828790 12061 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34969 } }
I20260620 14:59:25.829403 12061 leader_election.cc:302] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb977b6a846c4ed49d7dec39eb5c86a1; no voters:
I20260620 14:59:25.829980 12061 leader_election.cc:288] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:25.830201 12071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:25.831017 12071 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [term 1 LEADER]: Becoming Leader. State: Replica: cb977b6a846c4ed49d7dec39eb5c86a1, State: Running, Role: LEADER
I20260620 14:59:25.831591 12071 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34969 } }
I20260620 14:59:25.832537 12061 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:25.835029 12072 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34969 } } }
I20260620 14:59:25.839244 12072 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:25.835228 12073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb977b6a846c4ed49d7dec39eb5c86a1. Latest consensus state: current_term: 1 leader_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb977b6a846c4ed49d7dec39eb5c86a1" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 34969 } } }
I20260620 14:59:25.839958 12073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:25.842204 12077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:25.850251 12077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:25.860744 12077 catalog_manager.cc:1347] Generated new cluster ID: e0ef252c700f4891a08b1d34d937c721
I20260620 14:59:25.860877 12077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:25.876890 12077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:25.877537 12077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:25.885277 12077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb977b6a846c4ed49d7dec39eb5c86a1: Generated new TSK 0
I20260620 14:59:25.886497 12077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:25.898274 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11994
W20260620 14:59:25.899537 12056 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38309
W20260620 14:59:25.900318 22143 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 11994: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:59:25.900485 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 11994
2026-06-20T14:59:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2007 ms)
[----------] 18 tests from AuthTokenIssuingTest (37117 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:25 dist-test-slave-3mqx krb5kdc[12092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:25 dist-test-slave-3mqx krb5kdc[12092](info): set up 2 sockets
Jun 20 14:59:25 dist-test-slave-3mqx krb5kdc[12092](info): commencing operation
krb5kdc: starting...
W20260620 14:59:27.080139 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:27 dist-test-slave-3mqx krb5kdc[12092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:27.351246 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33827
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33827,127.21.159.253:43627,127.21.159.252:41945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:27.699297 12105 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:59:27.699698 12105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:27.699833 12105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:27.699987 12105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:27.707849 12105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:27.707948 12105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:27.708041 12105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:27.708119 12105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:27.708206 12105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:27.718932 12105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33827,127.21.159.253:43627,127.21.159.252:41945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:27.720393 12105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:27.721771 12105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:27.734480 12111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:27.734485 12113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:27.734776 12110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:27.735332 12105 server_base.cc:1034] running on GCE node
I20260620 14:59:27.736198 12105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:27.737733 12105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:27.738973 12105 hybrid_clock.cc:648] HybridClock initialized: now 1781967567738897 us; error 47 us; skew 500 ppm
I20260620 14:59:27.739419 12105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:27 dist-test-slave-3mqx krb5kdc[12092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:27.743043 12105 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:27.744834 12105 webserver.cc:460] Webserver started at http://127.21.159.254:42813/ using document root <none> and password file <none>
I20260620 14:59:27.745609 12105 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:27.745702 12105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:27.746073 12105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:27.749385 12105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "69fbfdeafc8240f2840d427f5ad27a6a"
format_stamp: "Formatted at 2026-06-20 14:59:27 on dist-test-slave-3mqx"
server_key: "90c8f2699a210baa3d27204b616725a4"
server_key_iv: "cda313e8ff2cc78a4ce19dbccb8438a8"
server_key_version: "encryptionkey@0"
I20260620 14:59:27.750286 12105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "69fbfdeafc8240f2840d427f5ad27a6a"
format_stamp: "Formatted at 2026-06-20 14:59:27 on dist-test-slave-3mqx"
server_key: "90c8f2699a210baa3d27204b616725a4"
server_key_iv: "cda313e8ff2cc78a4ce19dbccb8438a8"
server_key_version: "encryptionkey@0"
I20260620 14:59:27.756107 12105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 14:59:27.759904 12120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:27.761485 12105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:59:27.761756 12105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "69fbfdeafc8240f2840d427f5ad27a6a"
format_stamp: "Formatted at 2026-06-20 14:59:27 on dist-test-slave-3mqx"
server_key: "90c8f2699a210baa3d27204b616725a4"
server_key_iv: "cda313e8ff2cc78a4ce19dbccb8438a8"
server_key_version: "encryptionkey@0"
I20260620 14:59:27.762014 12105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:27.796351 12105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:27.801276 12105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:27.801594 12105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:27.821653 12105 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:33827
I20260620 14:59:27.821702 12171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:33827 every 8 connection(s)
I20260620 14:59:27.823204 12105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:27.827594 12172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:27.829337 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 12105
I20260620 14:59:27.829572 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:27.830233 22143 external_mini_cluster.cc:1428] Setting key bae2d843b00b2180170d0a614b4d0f8e
I20260620 14:59:27.835304 12172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:27 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.253@KRBTEST.COM" created.
I20260620 14:59:27.864320 12176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.839508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.21.159.254:33827)
0620 14:59:27.841785 (+ 2277us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:27.841792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:27.841856 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:59:27.844262 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:27.844268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:27.844571 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:27.844761 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:27.846628 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.848068 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.850531 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.851245 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.851415 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:27.856078 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:27.856092 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:27.856110 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:27.856158 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:27.859524 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:27.860582 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:27.860591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:27.860593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:27.860758 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:27.861285 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:27.861288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:27.861290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:27.861850 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:27.862013 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:27.862759 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 14:59:27.863145 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2033,"spinlock_wait_cycles":62208,"thread_start_us":1734,"threads_started":1}
I20260620 14:59:27.864348 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.839508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33827 (local address 127.0.0.1:54102)
0620 14:59:27.840557 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.840613 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:27.843740 (+ 3127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:27.844868 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:27.844878 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:27.845790 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:27.846348 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:27.846380 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.848254 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.848263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:27.850300 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:27.850322 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:27.851532 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:27.851543 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:27.851780 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:27.852553 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:27.852590 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:27.855758 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:27.859709 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:27.859720 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:27.859732 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:27.860417 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:27.860905 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:27.860912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:27.860918 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:27.861171 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:27.862029 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:27.862041 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:27.862346 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 14:59:27.862644 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 14:59:27.863363 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":147,"thread_start_us":441,"threads_started":1}
I20260620 14:59:27.867692 12172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:27.869027 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.868484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42324)
0620 14:59:27.868708 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.868844 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 14:59:27.869346 12123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
I20260620 14:59:27.871096 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.870652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42330)
0620 14:59:27.870816 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.870887 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:59:27.871317 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:43627: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab.
Entry for principal kudu/127.21.159.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab.
I20260620 14:59:27.894475 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:43627
--webserver_interface=127.21.159.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab
--principal=kudu/127.21.159.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33827,127.21.159.253:43627,127.21.159.252:41945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:59:27.920704 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } attempt: 1
I20260620 14:59:27.921916 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.921415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42340)
0620 14:59:27.921626 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.921785 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 14:59:27.922956 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.922587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42348)
0620 14:59:27.922738 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.922818 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:59:27.923163 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:43627: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
I20260620 14:59:27.991458 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } attempt: 2
I20260620 14:59:27.992817 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.992306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42364)
0620 14:59:27.992579 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.992675 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 14:59:27.993870 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:27.993484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42370)
0620 14:59:27.993658 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:27.993729 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:59:27.994062 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:43627: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
I20260620 14:59:28.085343 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } attempt: 3
I20260620 14:59:28.086526 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.086058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42380)
0620 14:59:28.086287 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.086387 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 14:59:28.087590 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.087204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42390)
0620 14:59:28.087356 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.087438 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:59:28.087764 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:43627: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
I20260620 14:59:28.231021 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } attempt: 4
I20260620 14:59:28.232373 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.231799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42396)
0620 14:59:28.232140 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.232231 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 14:59:28.233413 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.233048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42402)
0620 14:59:28.233212 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.233297 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:59:28.233654 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:43627: Network error: Client connection negotiation failed: client connection to 127.21.159.253:43627: connect: Connection refused (error 111)
W20260620 14:59:28.242913 12179 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:59:28.243398 12179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:28.243521 12179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:28.243641 12179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:28.251310 12179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:28.251416 12179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:28.251530 12179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:28.251616 12179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:28.251708 12179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:28.263397 12179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33827,127.21.159.253:43627,127.21.159.252:41945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab
--principal=kudu/127.21.159.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:43627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.159.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:28.264902 12179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:28.266523 12179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:28.280341 12187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.280489 12184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.280421 12185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:28.281176 12179 server_base.cc:1034] running on GCE node
I20260620 14:59:28.282033 12179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:28.283732 12179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:28.284991 12179 hybrid_clock.cc:648] HybridClock initialized: now 1781967568284920 us; error 51 us; skew 500 ppm
I20260620 14:59:28.285355 12179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:28.293372 12179 init.cc:377] Logged in from keytab as kudu/127.21.159.253@KRBTEST.COM (short username kudu)
I20260620 14:59:28.294997 12179 webserver.cc:460] Webserver started at http://127.21.159.253:42321/ using document root <none> and password file <none>
I20260620 14:59:28.295789 12179 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:28.295884 12179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:28.296171 12179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:28.299401 12179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data/instance:
uuid: "e26f27a256a44ea1bc387570979a9b64"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-3mqx"
server_key: "709c35c9c98d08f8b7c5c41aec1bfe39"
server_key_iv: "a87b41e53a53a1ea9d98ab7b937b6340"
server_key_version: "encryptionkey@0"
I20260620 14:59:28.300333 12179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/wal/instance:
uuid: "e26f27a256a44ea1bc387570979a9b64"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-3mqx"
server_key: "709c35c9c98d08f8b7c5c41aec1bfe39"
server_key_iv: "a87b41e53a53a1ea9d98ab7b937b6340"
server_key_version: "encryptionkey@0"
I20260620 14:59:28.306257 12179 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260620 14:59:28.310137 12194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.311806 12179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:59:28.312013 12179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/wal
uuid: "e26f27a256a44ea1bc387570979a9b64"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-3mqx"
server_key: "709c35c9c98d08f8b7c5c41aec1bfe39"
server_key_iv: "a87b41e53a53a1ea9d98ab7b937b6340"
server_key_version: "encryptionkey@0"
I20260620 14:59:28.312279 12179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:28.337502 12179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:28.342211 12179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:28.342514 12179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:28.362416 12179 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.253:43627
I20260620 14:59:28.362448 12245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.253:43627 every 8 connection(s)
I20260620 14:59:28.363945 12179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
I20260620 14:59:28.368674 12246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:28.371793 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 12179
I20260620 14:59:28.372008 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-1/wal/instance
I20260620 14:59:28.372524 22143 external_mini_cluster.cc:1428] Setting key 5ab61fe3e3a722d29defee30c631d413
I20260620 14:59:28.376572 12246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.252@KRBTEST.COM" created.
I20260620 14:59:28.403088 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.379517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.21.159.254:33827)
0620 14:59:28.380087 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.380096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.381286 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 14:59:28.383447 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.383454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.383566 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.383701 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.385873 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.386637 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.388450 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.388878 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.389043 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.393898 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.393909 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.393916 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.393990 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.398198 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.399773 (+ 1575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.399780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.399784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.399975 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.400422 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.400426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.400428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.400855 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.401049 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.402238 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.402541 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":278,"threads_started":1}
I20260620 14:59:28.404295 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.380256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33827 (local address 127.0.0.1:54110)
0620 14:59:28.381101 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.381144 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.383017 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.383992 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.384002 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.384775 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.385554 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.385580 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.386859 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.386863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.388265 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.388280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.389191 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.389203 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.389455 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.390256 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.390286 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.393489 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.398427 (+ 4938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.398442 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.398466 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.399546 (+ 1080us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.400110 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.400114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.400116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.400308 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.401088 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.401098 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.401722 (+ 624us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.402094 (+ 372us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.402699 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":333,"threads_started":1}
I20260620 14:59:28.406219 12246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:59:28.421635 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.407172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42406)
0620 14:59:28.407358 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.407377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.407497 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.408571 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.408576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.409003 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.409131 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.409139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.410541 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.410548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.411487 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.411498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.412342 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.412347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.412574 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.413211 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.413228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.415391 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.418915 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.418921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.418928 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.419438 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.419826 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.419833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.419836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.419988 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.420588 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.420598 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.420737 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.420889 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.421201 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":5}
I20260620 14:59:28.421667 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.407344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.21.159.253:43627)
0620 14:59:28.408039 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.408045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.408071 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:59:28.408117 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.408123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.408315 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.408481 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.409307 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.410406 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.411620 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.412189 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.412331 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.415626 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.415640 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.415654 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.415684 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.418752 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.419591 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.419598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.419600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.419715 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.420101 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.420112 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.420115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.420464 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.420692 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.421052 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.421312 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":542,"thread_start_us":400,"threads_started":1}
I20260620 14:59:28.425017 12246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:28.426409 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.425954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56356)
0620 14:59:28.426144 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.426251 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 14:59:28.426834 12196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
I20260620 14:59:28.428484 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.428122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56364)
0620 14:59:28.428289 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.428368 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:59:28.428757 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab.
Entry for principal kudu/127.21.159.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab.
I20260620 14:59:28.436868 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:41945
--webserver_interface=127.21.159.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab
--principal=kudu/127.21.159.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:33827,127.21.159.253:43627,127.21.159.252:41945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:59:28.478156 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 1
I20260620 14:59:28.479377 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.478862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56376)
0620 14:59:28.479091 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.479222 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 14:59:28.480433 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.480085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56390)
0620 14:59:28.480246 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.480324 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:59:28.480698 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
I20260620 14:59:28.532920 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } attempt: 5
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:59:28.547322 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.533699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42414)
0620 14:59:28.533896 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.533921 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.534059 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.534430 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.534437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.534882 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.535151 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.535163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.536221 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.536227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.537095 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.537106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.537877 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.537882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.538108 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.538784 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.538802 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.541342 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.544656 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.544662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.544666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.545099 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.545516 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.545522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.545525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.545674 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.546241 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.546245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.546409 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.546581 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.546935 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:59:28.547322 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.533816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.21.159.253:43627)
0620 14:59:28.534053 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.534059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.534083 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:59:28.534201 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.534214 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.534279 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.534390 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.535310 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.536054 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.537259 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.537710 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.537847 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.541615 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.541621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.541624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.541661 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.544453 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.545243 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.545249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.545255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.545390 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.545792 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.545798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.545802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.546114 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.546278 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.546692 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.546937 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 14:59:28.548539 12172 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:28.548909 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 2
I20260620 14:59:28.549722 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.549279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56404)
0620 14:59:28.549457 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.549557 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:59:28.549831 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.549428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56418)
0620 14:59:28.549593 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.549677 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 14:59:28.550761 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.550399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56420)
0620 14:59:28.550539 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.550627 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:59:28.550969 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
I20260620 14:59:28.551149 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.550701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56422)
0620 14:59:28.550876 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.550970 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 14:59:28.551314 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
I20260620 14:59:28.602267 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 1
I20260620 14:59:28.603472 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.602966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56430)
0620 14:59:28.603194 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.603302 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 14:59:28.604471 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.604134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56434)
0620 14:59:28.604286 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.604363 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:59:28.604713 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
I20260620 14:59:28.642560 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 3
I20260620 14:59:28.643980 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.643444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56440)
0620 14:59:28.643675 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.643778 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:59:28.645367 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.644837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56452)
0620 14:59:28.645130 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.645223 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260620 14:59:28.645597 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
I20260620 14:59:28.672935 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 2
I20260620 14:59:28.674130 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.673609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56458)
0620 14:59:28.673836 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.673942 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 14:59:28.675206 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.674803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56460)
0620 14:59:28.674957 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.675038 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:59:28.675410 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
I20260620 14:59:28.781617 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 3
I20260620 14:59:28.782872 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.782298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56472)
0620 14:59:28.782540 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.782663 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 14:59:28.784157 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.783664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56478)
0620 14:59:28.783914 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.784015 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 14:59:28.784389 12172 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
W20260620 14:59:28.788234 12254 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:59:28.788621 12254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:28.788731 12254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:28.788852 12254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:59:28.788833 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 4
I20260620 14:59:28.790006 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.789549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56488)
0620 14:59:28.789764 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.789861 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 14:59:28.791136 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.790722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56494)
0620 14:59:28.790888 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.790960 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:59:28.791334 12246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:41945: Network error: Client connection negotiation failed: client connection to 127.21.159.252:41945: connect: Connection refused (error 111)
W20260620 14:59:28.797091 12254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:28.797188 12254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:28.797292 12254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:28.797372 12254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:28.797459 12254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:28.807946 12254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:33827,127.21.159.253:43627,127.21.159.252:41945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab
--principal=kudu/127.21.159.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.159.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:28.809350 12254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:28.810745 12254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:28.824280 12260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.824262 12262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:28.825098 12259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:28.832113 12254 server_base.cc:1034] running on GCE node
I20260620 14:59:28.833096 12254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:28.834772 12254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:28.836071 12254 hybrid_clock.cc:648] HybridClock initialized: now 1781967568835982 us; error 41 us; skew 500 ppm
I20260620 14:59:28.836571 12254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:28.840441 12254 init.cc:377] Logged in from keytab as kudu/127.21.159.252@KRBTEST.COM (short username kudu)
I20260620 14:59:28.842147 12254 webserver.cc:460] Webserver started at http://127.21.159.252:46387/ using document root <none> and password file <none>
I20260620 14:59:28.842983 12254 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:28.843150 12254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:28.843509 12254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:28.846716 12254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data/instance:
uuid: "efe5868321c24fe3aed2fcba8393c4ff"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-3mqx"
server_key: "60077d7cab5ef9d49625249da7aeef6b"
server_key_iv: "6807d27d38a8bdd9f85108f488f168cd"
server_key_version: "encryptionkey@0"
I20260620 14:59:28.847693 12254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/wal/instance:
uuid: "efe5868321c24fe3aed2fcba8393c4ff"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-3mqx"
server_key: "60077d7cab5ef9d49625249da7aeef6b"
server_key_iv: "6807d27d38a8bdd9f85108f488f168cd"
server_key_version: "encryptionkey@0"
I20260620 14:59:28.853256 12254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:59:28.857077 12269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:28.858646 12254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:59:28.858870 12254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/wal
uuid: "efe5868321c24fe3aed2fcba8393c4ff"
format_stamp: "Formatted at 2026-06-20 14:59:28 on dist-test-slave-3mqx"
server_key: "60077d7cab5ef9d49625249da7aeef6b"
server_key_iv: "6807d27d38a8bdd9f85108f488f168cd"
server_key_version: "encryptionkey@0"
I20260620 14:59:28.859153 12254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:28.883425 12254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:28.887995 12254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:28.888309 12254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:28.907208 12254 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.252:41945
I20260620 14:59:28.907214 12320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.252:41945 every 8 connection(s)
I20260620 14:59:28.908663 12254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
I20260620 14:59:28.912663 12321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:28.914434 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 12254
I20260620 14:59:28.914645 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967200138920-22143-0/minicluster-data/master-2/wal/instance
I20260620 14:59:28.915241 22143 external_mini_cluster.cc:1428] Setting key 4a2d57568174d3febc0f0eb78d84c541
I20260620 14:59:28.922569 12321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:59:28.941030 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.917358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.21.159.252:41945)
0620 14:59:28.918232 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.918253 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.918304 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:59:28.920109 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.920139 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.920611 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.921036 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.922638 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.924166 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.925657 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.926845 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.927033 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.931973 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.931980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.931993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.932075 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.935875 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.936673 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.936682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.936694 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.936904 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.937415 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.937422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.937426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.938252 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.938566 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.939091 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.939582 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":379,"threads_started":1}
I20260620 14:59:28.942883 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.925399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:33827 (local address 127.0.0.1:54120)
0620 14:59:28.926276 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.926329 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.926491 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.927407 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.927415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.928320 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.928689 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.928704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.929871 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.929878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.931037 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.931115 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.932185 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.932192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.932405 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.933156 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.933181 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.935990 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.940053 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.940063 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.940068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.940514 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.940923 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.940926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.940928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.941082 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.941848 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.941855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.942049 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.942290 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.942587 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":417,"threads_started":1}
I20260620 14:59:28.943467 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.925390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.21.159.254:33827)
0620 14:59:28.926992 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.926999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.927017 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:59:28.927104 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.927110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.927190 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.927313 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.928917 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.929695 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.931275 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.931957 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.932073 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.936225 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.936230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.936234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.936276 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.939890 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.940651 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.940656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.940659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.940812 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.941386 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.941393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.941399 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.941704 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.941835 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.942410 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.942747 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":1284,"threads_started":1}
I20260620 14:59:28.944795 12321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:59:28.960264 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.945584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:43627 (local address 127.0.0.1:42424)
0620 14:59:28.945759 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.945778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.945886 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.946225 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.946230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.946706 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.946885 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.946896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.948074 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.948079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.949154 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.949168 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.950021 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.950027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.950200 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.950894 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.950931 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.953534 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.957219 (+ 3685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.957227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.957234 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.957843 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.958274 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.958279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.958283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.958488 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.959226 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.959230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.959414 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.959595 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.959895 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:59:28.960279 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.945607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.21.159.253:43627)
0620 14:59:28.945797 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.945805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.945825 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:59:28.946025 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.946030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.946100 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.946202 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.947124 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.947915 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.949313 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.949826 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.949959 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.953798 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.953808 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.953814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.953863 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.957017 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.957983 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.957988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.957990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.958148 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.958629 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.958633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.958636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.959008 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.959230 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.959716 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.959970 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:59:28.961505 12321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:28.961619 12172 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 4
I20260620 14:59:28.962292 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.944780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54132 (local address 127.21.159.254:33827)
0620 14:59:28.944984 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.944989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.945008 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:59:28.945088 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.945093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.945175 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.945287 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.946664 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.947554 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.948984 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.949561 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.949659 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.954763 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.954773 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.954781 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.954837 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.958480 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.959422 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.959428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.959431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.959595 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.960039 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.960045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.960049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.960415 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.960670 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.961270 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.961941 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
Jun 20 14:59:28 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:59:28.978549 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.962549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.21.159.252:41945)
0620 14:59:28.962917 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.962924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.962938 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:59:28.963007 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.963011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.963122 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.963231 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.964157 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.965338 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.967368 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.967872 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.968020 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.971751 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.971757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.971762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.971815 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.975297 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.976162 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.976169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.976172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.976310 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.976772 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.976783 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.976787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.977283 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.977426 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.977876 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.978122 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":287}
I20260620 14:59:28.978577 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.962194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56524)
0620 14:59:28.962414 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.962437 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.962533 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.963287 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.963292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.963840 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.964014 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.964022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.965540 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.965544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.967202 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.967220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.968048 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.968054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.968264 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.968940 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.968968 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.971530 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.975482 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.975488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.975493 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.976024 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.976500 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.976505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.976508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.976657 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.977398 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.977402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.977600 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.977764 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.978072 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 14:59:28.978876 12326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.962339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56536)
0620 14:59:28.962572 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:28.962589 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:28.962715 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:28.963624 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:28.963629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:28.964061 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:28.964324 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.964335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.965911 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.965918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.966976 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:28.966988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.968247 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.968253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:28.968497 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.969128 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:28.969150 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:28.972035 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:28.976298 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.976303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.976307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.976814 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.977183 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:28.977188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:28.977192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:28.977356 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:28.977965 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:28.977969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:28.978125 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:59:28.978256 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:59:28.978558 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":39}
I20260620 14:59:28.979727 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.962644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.21.159.252:41945)
0620 14:59:28.963326 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.963332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.963353 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:59:28.963398 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.963403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.963477 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.963584 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.964513 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.965793 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.967186 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.967922 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.968052 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.972266 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.972272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.972275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.972308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.976162 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.976933 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.976937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.976941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.977060 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.977489 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.977493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.977496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.977783 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.977908 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.979199 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.979344 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":685,"threads_started":1}
I20260620 14:59:28.984320 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:28.965064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.21.159.253:43627)
0620 14:59:28.965229 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:28.965235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:28.965277 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:59:28.965438 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:28.965443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:28.965516 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:28.965617 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:28.967358 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.968429 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.970293 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:28.971092 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:28.971228 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:28.977588 (+ 6360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:28.977594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:28.977599 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:28.977641 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:28.981185 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.982072 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.982078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.982082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.982252 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:28.982704 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:28.982709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:28.982712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:28.983100 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:28.983388 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:28.983709 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 14:59:28.983952 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:59:28.985550 12321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: Bootstrap starting.
I20260620 14:59:28.985569 12172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a: Bootstrap starting.
I20260620 14:59:28.989364 12172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:28.989384 12321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:28.989828 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:28.990585 12321 log.cc:824] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:28.990626 12172 log.cc:824] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:28.993450 12321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: No bootstrap required, opened a new log
I20260620 14:59:28.998396 12172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a: No bootstrap required, opened a new log
I20260620 14:59:28.994125 12147 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "69fbfdeafc8240f2840d427f5ad27a6a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54132
I20260620 14:59:29.001300 12321 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
I20260620 14:59:29.001716 12321 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe5868321c24fe3aed2fcba8393c4ff, State: Initialized, Role: FOLLOWER
I20260620 14:59:29.002318 12321 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
I20260620 14:59:29.003362 12330 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.003882 12330 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:29.004720 12321 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:29.005921 12172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
I20260620 14:59:29.006376 12172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fbfdeafc8240f2840d427f5ad27a6a, State: Initialized, Role: FOLLOWER
I20260620 14:59:29.006847 12172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
I20260620 14:59:29.007918 12334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.008343 12334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:29.009187 12172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:29.009338 12147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:59:29.009625 12147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:29.014137 12147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
I20260620 14:59:29.017370 12296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "efe5868321c24fe3aed2fcba8393c4ff"
I20260620 14:59:29.018031 12296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:29.022636 12296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fbfdeafc8240f2840d427f5ad27a6a in term 1.
I20260620 14:59:29.023592 12122 leader_election.cc:302] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fbfdeafc8240f2840d427f5ad27a6a, efe5868321c24fe3aed2fcba8393c4ff; no voters:
I20260620 14:59:29.024187 12334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:29.024982 12334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [term 1 LEADER]: Becoming Leader. State: Replica: 69fbfdeafc8240f2840d427f5ad27a6a, State: Running, Role: LEADER
I20260620 14:59:29.025091 12147 leader_election.cc:288] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [CANDIDATE]: Term 1 election: Requested vote from peers e26f27a256a44ea1bc387570979a9b64 (127.21.159.253:43627), efe5868321c24fe3aed2fcba8393c4ff (127.21.159.252:41945)
I20260620 14:59:29.025604 12334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
W20260620 14:59:29.047125 12351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:59:29.047355 12351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 14:59:29.054736 12126 tablet.cc:2369] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:59:29.054775 12354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69fbfdeafc8240f2840d427f5ad27a6a. Latest consensus state: current_term: 1 leader_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.058732 12354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:29.072937 12364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:29.074286 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.032074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.21.159.253:43627)
0620 14:59:29.032267 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.032275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.032606 (+ 331us) server_negotiation.cc:407] Connection header received
0620 14:59:29.033241 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.033251 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.033365 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.033476 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.039650 (+ 6174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.040913 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.045644 (+ 4731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.046365 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.046480 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.053838 (+ 7358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:29.053850 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:29.053858 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:29.053926 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:29.069397 (+ 15471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.070795 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.070801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.070805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.071016 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.071591 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.071596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.071599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.072066 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:29.073430 (+ 1364us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.073597 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.073890 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:59:29.076655 12364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:29.059226 12105 master_runner.cc:186] Error getting master registration for 127.21.159.253:43627: OK, instance_id { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" instance_seqno: 1781967568356788 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:59:29.077538 12105 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:59:29.080837 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.032033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.21.159.252:41945)
0620 14:59:29.032224 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.032232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.037659 (+ 5427us) server_negotiation.cc:407] Connection header received
0620 14:59:29.038107 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.038114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.038226 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.038306 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.042817 (+ 4511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.044007 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.047993 (+ 3986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.048801 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.048906 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.056181 (+ 7275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:29.056189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:29.056194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:29.056241 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:29.068018 (+ 11777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.073813 (+ 5795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.073820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.073824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.074137 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.076140 (+ 2003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.076179 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.076187 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.077220 (+ 1033us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:29.077796 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.079678 (+ 1882us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.080145 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:59:29.081156 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.035421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.21.159.254:33827)
0620 14:59:29.035625 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.035635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.040763 (+ 5128us) server_negotiation.cc:407] Connection header received
0620 14:59:29.041451 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.041458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.041568 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.041668 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.044438 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.045921 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.048202 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.049009 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.049145 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.057188 (+ 8043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:29.057194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:29.057198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:29.057243 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:29.070496 (+ 13253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.073111 (+ 2615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.073118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.073125 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.073464 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.078003 (+ 4539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.078010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.078013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.078528 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:29.078856 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.080450 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.080726 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:59:29.090909 12246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } attempt: 5
I20260620 14:59:29.085139 12254 master_runner.cc:186] Error getting master registration for 127.21.159.253:43627: OK, instance_id { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" instance_seqno: 1781967568356788 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:59:29.096237 12254 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:59:29.101756 12353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:59:29.101922 12353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:59:29.105949 12296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [term 1 FOLLOWER]: Refusing update from remote peer 69fbfdeafc8240f2840d427f5ad27a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 20 14:59:29 dist-test-slave-3mqx krb5kdc[12092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967568, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
I20260620 14:59:29.107167 12354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:59:29.120489 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.091553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:41945 (local address 127.0.0.1:56556)
0620 14:59:29.091918 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:29.091942 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:29.092080 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:29.093203 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:29.093209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:29.093759 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:29.093983 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:29.093994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.099113 (+ 5119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.099119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:29.100075 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:29.100088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.101154 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.101161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:29.101422 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.104530 (+ 3108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:29.104558 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:29.107698 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:29.116650 (+ 8952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:29.116657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:29.116661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:29.117317 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:29.118047 (+ 730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:29.118056 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:29.118060 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:29.118287 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:29.119313 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:29.119319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:29.119515 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 14:59:29.119687 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:59:29.120049 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260620 14:59:29.121889 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.092312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.21.159.252:41945)
0620 14:59:29.092621 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.092629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.092679 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:59:29.092776 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.092805 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.092949 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.094216 (+ 1267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.094480 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.098932 (+ 4452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.100254 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.100965 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.107929 (+ 6964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.108205 (+ 276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:29.108212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:29.108220 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:29.108284 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:29.116444 (+ 8160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.117636 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.117642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.117645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.117856 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:29.118456 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:29.118463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:29.118466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:29.118979 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:29.121007 (+ 2028us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.121184 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.121461 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 14:59:29.128728 12246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64: Bootstrap starting.
I20260620 14:59:29.133512 12246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:29.134999 12246 log.cc:824] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:29.135560 12330 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69fbfdeafc8240f2840d427f5ad27a6a. Latest consensus state: current_term: 1 leader_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.135905 12330 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:29.139542 12366 mvcc.cc:205] Tried to move back new op lower bound from 7298939163043934208 to 7298939162735157248. Current Snapshot: MvccSnapshot[applied={T|T < 7298939163043934208}]
I20260620 14:59:29.140291 12334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.140508 12367 mvcc.cc:205] Tried to move back new op lower bound from 7298939163043934208 to 7298939162735157248. Current Snapshot: MvccSnapshot[applied={T|T < 7298939163043934208}]
I20260620 14:59:29.140640 12334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:29.142634 12364 catalog_manager.cc:1347] Generated new cluster ID: 069d44a963124cc8bf298a72caf9876e
I20260620 14:59:29.142745 12364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:29.155633 12330 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.155951 12330 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:29.160559 12246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64: No bootstrap required, opened a new log
I20260620 14:59:29.166078 12364 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:29.168396 12364 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:29.178594 12246 raft_consensus.cc:348] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
I20260620 14:59:29.179162 12246 raft_consensus.cc:729] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26f27a256a44ea1bc387570979a9b64, State: Initialized, Role: FOLLOWER
I20260620 14:59:29.179839 12246 consensus_queue.cc:260] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } }
I20260620 14:59:29.183044 12364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a: Generated new TSK 0
I20260620 14:59:29.184360 12364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:29.191923 12372 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.192488 12372 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:29.198690 12246 sys_catalog.cc:564] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:29.198690 12221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e26f27a256a44ea1bc387570979a9b64"
I20260620 14:59:29.199303 12220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:29.203472 12220 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [term 1 FOLLOWER]: Refusing update from remote peer 69fbfdeafc8240f2840d427f5ad27a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:59:29.204738 12354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:59:29.236757 12221 raft_consensus.cc:2396] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 69fbfdeafc8240f2840d427f5ad27a6a for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 14:59:29.240142 12372 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69fbfdeafc8240f2840d427f5ad27a6a. Latest consensus state: current_term: 1 leader_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.240546 12372 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:29.241087 12377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.241650 12377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:29.322247 12392 catalog_manager.cc:1259] Loaded cluster ID: 069d44a963124cc8bf298a72caf9876e
I20260620 14:59:29.322371 12392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64: loading cluster ID for follower catalog manager: success
I20260620 14:59:29.338681 12392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64: acquiring CA information for follower catalog manager: success
I20260620 14:59:29.341681 12392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e26f27a256a44ea1bc387570979a9b64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:59:29.352468 12251 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.328458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.21.159.253:43627)
0620 14:59:29.328688 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.328696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.349444 (+ 20748us) server_negotiation.cc:407] Connection header received
0620 14:59:29.350610 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.350617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.350737 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.350822 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.352209 (+ 1387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42458: BlockingRecv error: recv got EOF from 127.0.0.1:42458 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 14:59:29.358166 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.332978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.21.159.252:41945)
0620 14:59:29.333198 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.333205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.354838 (+ 21633us) server_negotiation.cc:407] Connection header received
0620 14:59:29.356330 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.356337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.356440 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.356514 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.357863 (+ 1349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56572: BlockingRecv error: recv got EOF from 127.0.0.1:56572 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 14:59:29.360461 12325 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.344242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.21.159.254:33827)
0620 14:59:29.344478 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.344485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.355632 (+ 11147us) server_negotiation.cc:407] Connection header received
0620 14:59:29.358705 (+ 3073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.358712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.358807 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.358892 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.360197 (+ 1305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54138: BlockingRecv error: recv got EOF from 127.0.0.1:54138 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:59:29.409627 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.393165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.21.159.253:43627)
0620 14:59:29.393367 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.393385 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.399862 (+ 6477us) server_negotiation.cc:407] Connection header received
0620 14:59:29.400487 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.400494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.400729 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.400882 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.401736 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.402969 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.405427 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.406538 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.406647 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.407984 (+ 1337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.408741 (+ 757us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.408871 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.409123 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.409234 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 14:59:29.466323 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.452792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.21.159.252:41945)
0620 14:59:29.453045 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.453060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.463795 (+ 10735us) server_negotiation.cc:407] Connection header received
0620 14:59:29.464513 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.464520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.464673 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.464762 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.466039 (+ 1277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56574: BlockingRecv error: recv got EOF from 127.0.0.1:56574 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:59:29.524510 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.506175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.21.159.253:43627)
0620 14:59:29.506439 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.506447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.515230 (+ 8783us) server_negotiation.cc:407] Connection header received
0620 14:59:29.515925 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.515932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.516065 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.516142 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.517073 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.518617 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.521160 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.521985 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.522117 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.523422 (+ 1305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.523637 (+ 215us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.523716 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.523980 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.524086 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 14:59:29.575083 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.559870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.21.159.252:41945)
0620 14:59:29.560085 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.560092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.572887 (+ 12795us) server_negotiation.cc:407] Connection header received
0620 14:59:29.573466 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.573473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.573583 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.573662 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.574771 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56580: BlockingRecv error: recv got EOF from 127.0.0.1:56580 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:59:29.635393 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.621033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.21.159.253:43627)
0620 14:59:29.621261 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.621268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.627810 (+ 6542us) server_negotiation.cc:407] Connection header received
0620 14:59:29.628038 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.628052 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.628158 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.628281 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.629096 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.630375 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.632697 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.633520 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.633658 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.634404 (+ 746us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.634564 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.634631 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.634868 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.634980 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:59:29.680166 12354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fbfdeafc8240f2840d427f5ad27a6a" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 33827 } } peers { permanent_uuid: "e26f27a256a44ea1bc387570979a9b64" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 43627 } } peers { permanent_uuid: "efe5868321c24fe3aed2fcba8393c4ff" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 41945 } } }
I20260620 14:59:29.680714 12354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69fbfdeafc8240f2840d427f5ad27a6a [sys.catalog]: This master's current role is: LEADER
W20260620 14:59:29.701395 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.688128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.21.159.252:41945)
0620 14:59:29.688327 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.688333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.699178 (+ 10845us) server_negotiation.cc:407] Connection header received
0620 14:59:29.699809 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.699816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.699930 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.700017 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.701143 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56582: BlockingRecv error: recv got EOF from 127.0.0.1:56582 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:59:29.763734 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.745787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.21.159.253:43627)
0620 14:59:29.746027 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.746034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.755868 (+ 9834us) server_negotiation.cc:407] Connection header received
0620 14:59:29.756591 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.756598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.756707 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.756822 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.757537 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.758757 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.760958 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.761666 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.761790 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.762619 (+ 829us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.762818 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.762893 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.763291 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.763392 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 14:59:29.774996 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.772377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.21.159.252:41945)
0620 14:59:29.772659 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.772671 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.773258 (+ 587us) server_negotiation.cc:407] Connection header received
0620 14:59:29.773488 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.773494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.773600 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.773733 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.774702 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56596: BlockingRecv error: recv got EOF from 127.0.0.1:56596 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20260620 14:59:29.795363 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.787568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.21.159.253:43627)
0620 14:59:29.787785 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.787793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.788392 (+ 599us) server_negotiation.cc:407] Connection header received
0620 14:59:29.788642 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.788653 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.788750 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.788860 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.789416 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.790663 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.792888 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.793635 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.793802 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.794366 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.794545 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.794654 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.794866 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.794970 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 14:59:29.809753 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.807223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.21.159.252:41945)
0620 14:59:29.807456 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.807463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.808025 (+ 562us) server_negotiation.cc:407] Connection header received
0620 14:59:29.808257 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.808263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.808366 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.808495 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.809445 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56602: BlockingRecv error: recv got EOF from 127.0.0.1:56602 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:59:29.860152 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.852155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.21.159.253:43627)
0620 14:59:29.852393 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.852401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.853026 (+ 625us) server_negotiation.cc:407] Connection header received
0620 14:59:29.853262 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.853268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.853381 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.853501 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.854086 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.855354 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.857575 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.858314 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.858485 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.859186 (+ 701us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.859369 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.859475 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.859691 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.859802 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 14:59:29.877574 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.868852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.21.159.252:41945)
0620 14:59:29.869083 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.869092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.875988 (+ 6896us) server_negotiation.cc:407] Connection header received
0620 14:59:29.876189 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.876196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.876304 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.876447 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.877309 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56604: BlockingRecv error: recv got EOF from 127.0.0.1:56604 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:59:29.924494 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:29.916784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.21.159.253:43627)
0620 14:59:29.917032 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.917040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.917657 (+ 617us) server_negotiation.cc:407] Connection header received
0620 14:59:29.917868 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.917875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.918001 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.918125 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:29.918576 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.919990 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.922112 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:29.922901 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:29.923087 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:29.923579 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:29.923727 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:29.923827 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:29.924033 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 14:59:29.924132 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 14:59:29.939265 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:29.937079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.21.159.252:41945)
0620 14:59:29.937344 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:29.937351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:29.937617 (+ 266us) server_negotiation.cc:407] Connection header received
0620 14:59:29.937814 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:29.937819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:29.937916 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:29.938016 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:29.938936 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56608: BlockingRecv error: recv got EOF from 127.0.0.1:56608 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 14:59:30.013028 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:30.007212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42522 (local address 127.21.159.253:43627)
0620 14:59:30.007474 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:30.007483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:30.007825 (+ 342us) server_negotiation.cc:407] Connection header received
0620 14:59:30.008087 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:30.008091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:30.008164 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:30.008271 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:30.008744 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.009631 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.011168 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.011640 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.011804 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:30.012263 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:30.012393 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:30.012498 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:30.012658 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 14:59:30.012747 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 14:59:30.021803 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:30.019163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.21.159.252:41945)
0620 14:59:30.019415 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:30.019426 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:30.019926 (+ 500us) server_negotiation.cc:407] Connection header received
0620 14:59:30.020524 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:30.020528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:30.020691 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:30.020785 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:30.021550 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56610: BlockingRecv error: recv got EOF from 127.0.0.1:56610 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:59:30.104120 12353 catalog_manager.cc:1259] Loaded cluster ID: 069d44a963124cc8bf298a72caf9876e
I20260620 14:59:30.104266 12353 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: loading cluster ID for follower catalog manager: success
I20260620 14:59:30.107656 12353 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: acquiring CA information for follower catalog manager: success
I20260620 14:59:30.109658 12353 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efe5868321c24fe3aed2fcba8393c4ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:59:30.159607 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:30.154148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42538 (local address 127.21.159.253:43627)
0620 14:59:30.154414 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:30.154424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:30.154630 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:59:30.154883 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:30.154887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:30.154960 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:30.155092 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:30.155559 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.156291 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.157771 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.158222 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.158368 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:30.158800 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:30.158919 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:30.159024 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:30.159227 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 14:59:30.159316 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 14:59:30.172297 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:30.165974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.21.159.252:41945)
0620 14:59:30.166247 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:30.166254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:30.166695 (+ 441us) server_negotiation.cc:407] Connection header received
0620 14:59:30.166964 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:30.166968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:30.167194 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:30.167322 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:30.167764 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.168526 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.169871 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.170299 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.170442 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:30.171010 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:30.171640 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:30.171807 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:30.171887 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:59:30.171980 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:59:30.190011 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:30.178936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.21.159.252:41945)
0620 14:59:30.179207 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:30.179217 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:30.183243 (+ 4026us) server_negotiation.cc:407] Connection header received
0620 14:59:30.183649 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:30.183656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:30.183785 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:30.183855 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:30.184742 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.185767 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.187930 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.188595 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.188718 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:30.189231 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:30.189352 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:30.189454 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:30.189615 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:59:30.189714 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:59:30.190508 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:30.179035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.21.159.253:43627)
0620 14:59:30.179342 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:30.179352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:30.184014 (+ 4662us) server_negotiation.cc:407] Connection header received
0620 14:59:30.184223 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:30.184231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:30.184323 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:30.184429 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:30.185215 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.186303 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.188468 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.189080 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.189205 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:30.189806 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:30.189931 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:30.189996 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:30.190228 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:59:30.190316 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 14:59:30.191561 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:30.178847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.21.159.254:33827)
0620 14:59:30.182778 (+ 3931us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:30.182787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:30.183735 (+ 948us) server_negotiation.cc:407] Connection header received
0620 14:59:30.184075 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:30.184083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:30.184321 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:30.184516 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:30.185479 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.186532 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.188357 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:30.188891 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:30.189020 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:30.189646 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:30.190138 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:30.190625 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:30.190854 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:59:30.190945 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3749,"thread_start_us":3640,"threads_started":1}
I20260620 14:59:30.196435 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 12105
I20260620 14:59:30.198669 12167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:30.314736 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 12105
I20260620 14:59:30.336707 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 12179
I20260620 14:59:30.338035 12241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:30.455327 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 12179
I20260620 14:59:30.473869 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 12254
I20260620 14:59:30.475140 12316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:30.585436 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 12254
2026-06-20T14:59:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4692 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:30 dist-test-slave-3mqx krb5kdc[12502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:30 dist-test-slave-3mqx krb5kdc[12502](info): set up 2 sockets
Jun 20 14:59:30 dist-test-slave-3mqx krb5kdc[12502](info): commencing operation
krb5kdc: starting...
W20260620 14:59:31.719830 22143 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/test-user.keytab.
Jun 20 14:59:31 dist-test-slave-3mqx krb5kdc[12502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.159.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
Entry for principal kudu/127.21.159.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab.
I20260620 14:59:32.003988 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46163
--webserver_interface=127.21.159.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:46163,127.21.159.253:40139,127.21.159.252:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:32.359680 12515 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:59:32.360122 12515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:32.360251 12515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:32.360415 12515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:32.368116 12515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:32.368256 12515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:32.368366 12515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:32.368460 12515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:32.368551 12515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:32.378943 12515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:46163,127.21.159.253:40139,127.21.159.252:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.254.keytab
--principal=kudu/127.21.159.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.254:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.159.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:32.380486 12515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:32.381898 12515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:32.396389 12521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:32.396389 12520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:32.398108 12523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:32.399781 12515 server_base.cc:1034] running on GCE node
I20260620 14:59:32.400959 12515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:32.403297 12515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:32.404692 12515 hybrid_clock.cc:648] HybridClock initialized: now 1781967572404506 us; error 56 us; skew 500 ppm
I20260620 14:59:32.405131 12515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:32 dist-test-slave-3mqx krb5kdc[12502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:32.409629 12515 init.cc:377] Logged in from keytab as kudu/127.21.159.254@KRBTEST.COM (short username kudu)
I20260620 14:59:32.411427 12515 webserver.cc:460] Webserver started at http://127.21.159.254:37285/ using document root <none> and password file <none>
I20260620 14:59:32.412520 12515 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:32.412681 12515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:32.413086 12515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:32.418000 12515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data/instance:
uuid: "d56abbca7b0b497594678803515a8785"
format_stamp: "Formatted at 2026-06-20 14:59:32 on dist-test-slave-3mqx"
server_key: "125565440311c18cdd5632e8b931ab3c"
server_key_iv: "3606c83fdc9c98e3eb6b388f71692e30"
server_key_version: "encryptionkey@0"
I20260620 14:59:32.419335 12515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/wal/instance:
uuid: "d56abbca7b0b497594678803515a8785"
format_stamp: "Formatted at 2026-06-20 14:59:32 on dist-test-slave-3mqx"
server_key: "125565440311c18cdd5632e8b931ab3c"
server_key_iv: "3606c83fdc9c98e3eb6b388f71692e30"
server_key_version: "encryptionkey@0"
I20260620 14:59:32.426646 12515 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 14:59:32.430765 12530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:32.432581 12515 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:59:32.432808 12515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/wal
uuid: "d56abbca7b0b497594678803515a8785"
format_stamp: "Formatted at 2026-06-20 14:59:32 on dist-test-slave-3mqx"
server_key: "125565440311c18cdd5632e8b931ab3c"
server_key_iv: "3606c83fdc9c98e3eb6b388f71692e30"
server_key_version: "encryptionkey@0"
I20260620 14:59:32.433050 12515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:32.457738 12515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:32.462939 12515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:32.463328 12515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:32.483539 12515 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.254:46163
I20260620 14:59:32.483587 12581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.254:46163 every 8 connection(s)
I20260620 14:59:32.485019 12515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:32.489176 12582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:32.493324 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 12515
I20260620 14:59:32.493533 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-0/wal/instance
I20260620 14:59:32.494092 22143 external_mini_cluster.cc:1428] Setting key 387f4f6e293beba6f77c18c2931b8116
I20260620 14:59:32.498467 12582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:32 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.253@KRBTEST.COM" created.
I20260620 14:59:32.525640 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.502981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.21.159.254:46163)
0620 14:59:32.504976 (+ 1995us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:32.504989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:32.505055 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:59:32.506738 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:32.506744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:32.507036 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:32.507338 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:32.508753 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:32.509987 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:32.511553 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:32.512150 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:32.512384 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:32.516786 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:32.516796 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:32.516810 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:32.516842 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:32.520143 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:32.521202 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:32.521209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:32.521211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:32.521435 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:32.521876 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:32.521881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:32.521884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:32.522627 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:32.522833 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:32.523561 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 14:59:32.523999 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":1756,"thread_start_us":1447,"threads_started":1}
I20260620 14:59:32.525655 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.502981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46163 (local address 127.0.0.1:43002)
0620 14:59:32.503968 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.504021 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:32.506269 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:32.507355 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:32.507361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:32.508154 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:32.508507 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:32.508526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:32.510169 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:32.510174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:32.511402 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:32.511416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:32.512428 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:32.512435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:32.512598 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:32.513346 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:32.513372 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:32.516465 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:32.520336 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:32.520355 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:32.520371 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:32.521046 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:32.521562 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:32.521567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:32.521571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:32.521749 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:32.522766 (+ 1017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:32.522774 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:32.523006 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 14:59:32.523351 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 14:59:32.524079 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":219,"thread_start_us":399,"threads_started":1}
I20260620 14:59:32.529165 12582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:32.530515 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.529921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53870)
0620 14:59:32.530160 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.530312 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 14:59:32.530848 12532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
I20260620 14:59:32.532721 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.532241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53886)
0620 14:59:32.532427 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.532523 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:59:32.533069 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:40139: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab.
Entry for principal kudu/127.21.159.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab.
I20260620 14:59:32.558642 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:40139
--webserver_interface=127.21.159.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab
--principal=kudu/127.21.159.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:46163,127.21.159.253:40139,127.21.159.252:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:59:32.582472 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } attempt: 1
I20260620 14:59:32.583854 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.583363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53892)
0620 14:59:32.583578 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.583718 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 14:59:32.585022 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.584671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53908)
0620 14:59:32.584804 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.584879 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:59:32.585314 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:40139: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
I20260620 14:59:32.653622 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } attempt: 2
I20260620 14:59:32.654974 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.654466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53920)
0620 14:59:32.654725 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.654826 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 14:59:32.656155 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.655777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53928)
0620 14:59:32.655921 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.655996 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:59:32.656422 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:40139: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
I20260620 14:59:32.747725 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } attempt: 3
I20260620 14:59:32.749076 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.748580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53938)
0620 14:59:32.748813 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.748923 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 14:59:32.750289 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.749873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53948)
0620 14:59:32.750061 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.750135 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:59:32.750581 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:40139: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
I20260620 14:59:32.893900 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } attempt: 4
I20260620 14:59:32.895330 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.894724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53962)
0620 14:59:32.895098 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.895190 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 14:59:32.896504 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:32.896112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53974)
0620 14:59:32.896277 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:32.896366 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 14:59:32.896796 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.253:40139: Network error: Client connection negotiation failed: client connection to 127.21.159.253:40139: connect: Connection refused (error 111)
W20260620 14:59:32.915244 12589 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:59:32.915656 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:32.915786 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:32.915938 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:32.923668 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:32.923784 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:32.923842 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:32.923942 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:32.924053 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:32.935007 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:46163,127.21.159.253:40139,127.21.159.252:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.253.keytab
--principal=kudu/127.21.159.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.253:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.159.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:32.936460 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:32.938055 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:32.951471 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:32.951380 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:32.951426 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:32.956919 12589 server_base.cc:1034] running on GCE node
I20260620 14:59:32.957844 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:32.959604 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:32.960861 12589 hybrid_clock.cc:648] HybridClock initialized: now 1781967572960791 us; error 40 us; skew 500 ppm
I20260620 14:59:32.961205 12589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:32 dist-test-slave-3mqx krb5kdc[12502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:32.964895 12589 init.cc:377] Logged in from keytab as kudu/127.21.159.253@KRBTEST.COM (short username kudu)
I20260620 14:59:32.966737 12589 webserver.cc:460] Webserver started at http://127.21.159.253:45161/ using document root <none> and password file <none>
I20260620 14:59:32.967648 12589 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:32.967788 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:32.968147 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:32.971506 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data/instance:
uuid: "566e0643eae54a79be278b50b8c22dd5"
format_stamp: "Formatted at 2026-06-20 14:59:32 on dist-test-slave-3mqx"
server_key: "1b24501c178421369160c29d74a8ec42"
server_key_iv: "0c298cc8f640ab941b9c247deb7c2633"
server_key_version: "encryptionkey@0"
I20260620 14:59:32.972409 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/wal/instance:
uuid: "566e0643eae54a79be278b50b8c22dd5"
format_stamp: "Formatted at 2026-06-20 14:59:32 on dist-test-slave-3mqx"
server_key: "1b24501c178421369160c29d74a8ec42"
server_key_iv: "0c298cc8f640ab941b9c247deb7c2633"
server_key_version: "encryptionkey@0"
I20260620 14:59:32.978255 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:59:32.982503 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:32.984162 12589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:59:32.984400 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/wal
uuid: "566e0643eae54a79be278b50b8c22dd5"
format_stamp: "Formatted at 2026-06-20 14:59:32 on dist-test-slave-3mqx"
server_key: "1b24501c178421369160c29d74a8ec42"
server_key_iv: "0c298cc8f640ab941b9c247deb7c2633"
server_key_version: "encryptionkey@0"
I20260620 14:59:32.984694 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:33.002089 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:33.007120 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:33.007455 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:33.028162 12589 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.253:40139
I20260620 14:59:33.028192 12655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.253:40139 every 8 connection(s)
I20260620 14:59:33.029729 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/data/info.pb
I20260620 14:59:33.034022 12656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:33.036806 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 12589
I20260620 14:59:33.037034 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-1/wal/instance
I20260620 14:59:33.037570 22143 external_mini_cluster.cc:1428] Setting key 310e7a363dae0b1cbb4ae8b75e82c668
I20260620 14:59:33.042012 12656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.159.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.159.252@KRBTEST.COM" created.
I20260620 14:59:33.070181 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.045244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.21.159.254:46163)
0620 14:59:33.045867 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.045875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.047106 (+ 1231us) server_negotiation.cc:407] Connection header received
0620 14:59:33.050571 (+ 3465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.050576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.050666 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.050786 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.053486 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.054287 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.057206 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.057701 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.057838 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.063207 (+ 5369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.063212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.063216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.063274 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.066324 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.067408 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.067414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.067418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.067549 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.068070 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.068074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.068077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.068368 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.068554 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.069666 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.069875 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":310,"threads_started":1}
I20260620 14:59:33.071873 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.046000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46163 (local address 127.0.0.1:43014)
0620 14:59:33.046860 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.046916 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.050057 (+ 3141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.051220 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.051237 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.052176 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.053132 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.053166 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.054522 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.054531 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.056918 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.056946 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.058035 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.058046 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.058455 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.059484 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.059525 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.062797 (+ 3272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.066546 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.066557 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.066596 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.067243 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.067682 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.067687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.067691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.067925 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.068560 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.068571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.069152 (+ 581us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.069507 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.070169 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":365,"threads_started":1}
I20260620 14:59:33.073979 12656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:59:33.091161 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.074890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:53990)
0620 14:59:33.075133 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.075155 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.075259 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.076435 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.076440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.076954 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.077088 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.077097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.078545 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.078553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.079556 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.079569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.080445 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.080450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.080691 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.081402 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.081420 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.084214 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.087979 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.087988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.087994 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.088574 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.089149 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.089152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.089155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.089338 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.089989 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.089994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.090170 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.090309 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.090683 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":5}
I20260620 14:59:33.091224 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.075237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.21.159.253:40139)
0620 14:59:33.075958 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.075964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.075990 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:59:33.076047 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.076053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.076266 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.076454 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.077292 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.078394 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.079700 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.080280 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.080432 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.084493 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.084512 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.084529 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.084563 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.087772 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.088729 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.088737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.088740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.088870 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.089463 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.089467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.089469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.089859 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.090086 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.090455 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.090808 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":566,"thread_start_us":418,"threads_started":1}
I20260620 14:59:33.094098 12656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:33.095538 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.094869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59774)
0620 14:59:33.095158 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.095321 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 14:59:33.095821 12607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.097481 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.097078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59788)
0620 14:59:33.097275 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.097352 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 14:59:33.097726 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.159.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab.
Entry for principal kudu/127.21.159.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab.
I20260620 14:59:33.106519 22143 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
/tmp/dist-test-taskmvfD4B/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:34375
--webserver_interface=127.21.159.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab
--principal=kudu/127.21.159.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.159.212:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.159.254:46163,127.21.159.253:40139,127.21.159.252:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:59:33.147149 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 1
I20260620 14:59:33.148440 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.147945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59796)
0620 14:59:33.148161 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.148275 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:59:33.149660 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.149237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59804)
0620 14:59:33.149425 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.149508 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 14:59:33.149984 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.196081 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } attempt: 5
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:59:33.215831 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.196999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:54000)
0620 14:59:33.197248 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.197275 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.197447 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.198422 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.198427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.198970 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.199281 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.199296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.200710 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.200721 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.202377 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.202401 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.203527 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.203537 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.203843 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.204581 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.204601 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.207256 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.211784 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.211795 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.211802 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.212598 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.213098 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.213105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.213112 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.213330 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.214309 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.214317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.214563 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.214773 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.215255 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:59:33.217170 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.197741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.21.159.253:40139)
0620 14:59:33.197946 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.197953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.197973 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:59:33.198040 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.198045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.198123 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.198223 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.199503 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.200540 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.202598 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.203323 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.203463 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.207538 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.207548 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.207556 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.207613 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.211562 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.212752 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.212758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.212761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.212939 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.213531 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.213540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.213543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.214040 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.214226 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.216359 (+ 2133us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.216705 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:59:33.218214 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 2
I20260620 14:59:33.219436 12582 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:33.219647 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.218941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59816)
0620 14:59:33.219215 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.219351 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:59:33.220994 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.220519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59832)
0620 14:59:33.220758 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.220847 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 14:59:33.221176 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.220408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59818)
0620 14:59:33.220777 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.220892 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
W20260620 14:59:33.221274 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.222505 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.221944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59838)
0620 14:59:33.222197 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.222316 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260620 14:59:33.222774 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.274072 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 1
I20260620 14:59:33.275404 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.274815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59848)
0620 14:59:33.275125 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.275219 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 14:59:33.276633 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.276234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59850)
0620 14:59:33.276403 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.276510 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 14:59:33.276955 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.312537 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 3
I20260620 14:59:33.313918 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.313328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59852)
0620 14:59:33.313575 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.313711 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 14:59:33.315176 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.314634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59862)
0620 14:59:33.314905 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.314986 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260620 14:59:33.315469 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.345257 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 2
I20260620 14:59:33.346699 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.346168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59864)
0620 14:59:33.346426 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.346525 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 14:59:33.347873 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.347510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59866)
0620 14:59:33.347668 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.347752 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:59:33.348104 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.454335 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 3
I20260620 14:59:33.455646 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.455166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59868)
0620 14:59:33.455365 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.455469 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:59:33.456683 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.456316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59882)
0620 14:59:33.456499 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.456584 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 14:59:33.456884 12582 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
I20260620 14:59:33.458691 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 4
I20260620 14:59:33.459852 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.459293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59890)
0620 14:59:33.459554 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.459674 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 14:59:33.460989 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.460622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59900)
0620 14:59:33.460794 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.460873 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 14:59:33.461388 12656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.159.252:34375: Network error: Client connection negotiation failed: client connection to 127.21.159.252:34375: connect: Connection refused (error 111)
W20260620 14:59:33.479506 12664 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:59:33.479928 12664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:33.480058 12664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:33.480197 12664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:33.488600 12664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:33.488729 12664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:33.488837 12664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:33.488922 12664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:33.489027 12664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:33.500195 12664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.159.212:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.159.254:46163,127.21.159.253:40139,127.21.159.252:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/krb5kdc/kudu_127.21.159.252.keytab
--principal=kudu/127.21.159.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.159.252:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.159.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:33.501691 12664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:33.503247 12664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:33.515931 12669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:33.515901 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:33.515978 12670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:33.516819 12664 server_base.cc:1034] running on GCE node
I20260620 14:59:33.517673 12664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:33.519315 12664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:33.520565 12664 hybrid_clock.cc:648] HybridClock initialized: now 1781967573520390 us; error 156 us; skew 500 ppm
I20260620 14:59:33.520944 12664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:33.528010 12664 init.cc:377] Logged in from keytab as kudu/127.21.159.252@KRBTEST.COM (short username kudu)
I20260620 14:59:33.529784 12664 webserver.cc:460] Webserver started at http://127.21.159.252:37655/ using document root <none> and password file <none>
I20260620 14:59:33.530550 12664 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:33.530696 12664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:33.531035 12664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:33.534230 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data/instance:
uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f"
format_stamp: "Formatted at 2026-06-20 14:59:33 on dist-test-slave-3mqx"
server_key: "cd5b8754634962637afe9fcc9a267afd"
server_key_iv: "0d63b7ef3144a53982c9b33fc577d7ce"
server_key_version: "encryptionkey@0"
I20260620 14:59:33.535159 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/wal/instance:
uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f"
format_stamp: "Formatted at 2026-06-20 14:59:33 on dist-test-slave-3mqx"
server_key: "cd5b8754634962637afe9fcc9a267afd"
server_key_iv: "0d63b7ef3144a53982c9b33fc577d7ce"
server_key_version: "encryptionkey@0"
I20260620 14:59:33.540735 12664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 14:59:33.544665 12679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:33.546293 12664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:59:33.546501 12664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data,/tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/wal
uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f"
format_stamp: "Formatted at 2026-06-20 14:59:33 on dist-test-slave-3mqx"
server_key: "cd5b8754634962637afe9fcc9a267afd"
server_key_iv: "0d63b7ef3144a53982c9b33fc577d7ce"
server_key_version: "encryptionkey@0"
I20260620 14:59:33.546761 12664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:33.593422 12664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:33.598354 12664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:33.598687 12664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:33.618824 12664 rpc_server.cc:307] RPC server started. Bound to: 127.21.159.252:34375
I20260620 14:59:33.618841 12730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.159.252:34375 every 8 connection(s)
I20260620 14:59:33.620378 12664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/data/info.pb
I20260620 14:59:33.624593 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:33.625478 22143 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu as pid 12664
I20260620 14:59:33.625684 22143 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskmvfD4B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967200138920-22143-0/minicluster-data/master-2/wal/instance
I20260620 14:59:33.626159 22143 external_mini_cluster.cc:1428] Setting key e771ad7e4963484950d4b5e6b00c50d7
I20260620 14:59:33.634105 12582 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 4
I20260620 14:59:33.635964 12731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.254@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
I20260620 14:59:33.655156 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.634878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59920)
0620 14:59:33.635127 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.635150 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.635271 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.636288 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.636294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.636848 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.637052 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.637065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.638782 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.638788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.640566 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.640591 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.642353 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.642363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.642648 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.643387 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.643409 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.646172 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.651615 (+ 5443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.651620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.651625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.652241 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.652714 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.652718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.652722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.652888 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.653969 (+ 1081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.653976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.654181 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.654320 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.654642 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
I20260620 14:59:33.656759 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.628306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.21.159.252:34375)
0620 14:59:33.629201 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.629218 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.629281 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:59:33.632079 (+ 2798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.632116 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.632788 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.633321 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.634951 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.637003 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.638999 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.640446 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.640723 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.647399 (+ 6676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.647408 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.647414 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.647465 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.651467 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.652300 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.652310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.652319 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.652504 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.653003 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.653009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.653018 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.653774 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.654141 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.654635 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.655178 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":918,"server-negotiator.queue_time_us":600,"spinlock_wait_cycles":36992,"thread_start_us":361,"threads_started":1}
I20260620 14:59:33.656796 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.635217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.21.159.252:34375)
0620 14:59:33.635907 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.635918 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.635948 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:59:33.636000 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.636009 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.636085 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.636203 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.637291 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.638594 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.640811 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.641884 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.642048 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.646468 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.646477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.646489 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.646538 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.651351 (+ 4813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.652387 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.652392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.652397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.652585 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.653031 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.653039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.653045 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.653781 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.655178 (+ 1397us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.655306 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.655499 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":520,"thread_start_us":307,"threads_started":1}
I20260620 14:59:33.664052 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.639333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.254:46163 (local address 127.0.0.1:43026)
0620 14:59:33.645119 (+ 5786us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.645180 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.645359 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.646109 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.646117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.647193 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.647765 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.647786 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.649356 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.649361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.651316 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.651330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.652608 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.652616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.652801 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.653486 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.653514 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.656270 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.660986 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.660999 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.661007 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.661473 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.661901 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.661907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.661911 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.662062 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.662744 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.662753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.662952 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.663203 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.663596 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5491,"thread_start_us":3878,"threads_started":1}
I20260620 14:59:33.664206 12736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.639301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.21.159.254:46163)
0620 14:59:33.645641 (+ 6340us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.645652 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.645672 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:59:33.645729 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.645734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.645819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.645931 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.647956 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.649147 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.651474 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.652175 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.652314 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.656499 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.656504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.656507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.656538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.660847 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.661615 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.661621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.661637 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.661782 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.662190 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.662195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.662198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.662569 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.662777 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.663545 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.663791 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6176,"thread_start_us":5994,"threads_started":1}
I20260620 14:59:33.665731 12731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:33.666474 12582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785: Bootstrap starting.
I20260620 14:59:33.670574 12582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:33.671934 12582 log.cc:824] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.254@KRBTEST.COM
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
I20260620 14:59:33.675921 12582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785: No bootstrap required, opened a new log
I20260620 14:59:33.681818 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.660137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.21.159.254:46163)
0620 14:59:33.663595 (+ 3458us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.663601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.663633 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:59:33.663684 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.663689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.663767 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.664735 (+ 968us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.665714 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.666945 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.668859 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.669361 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.669501 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.675013 (+ 5512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.675021 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.675026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.675093 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.678550 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.679465 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.679478 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.679480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.679678 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.680129 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.680135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.680137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.680458 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.680782 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.681099 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.681347 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3307,"thread_start_us":3110,"threads_started":1}
I20260620 14:59:33.683020 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.666697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.21.159.253:40139)
0620 14:59:33.666918 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.666923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.667370 (+ 447us) server_negotiation.cc:407] Connection header received
0620 14:59:33.667520 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.667526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.667606 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.667711 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.668615 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.669600 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.671297 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.671983 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.672122 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.676029 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.676036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.676040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.676084 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.679683 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.680665 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.680670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.680673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.680838 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.681318 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.681325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.681329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.681687 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.681820 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.682381 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.682634 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:59:33.684083 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.666469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.253:40139 (local address 127.0.0.1:54010)
0620 14:59:33.667268 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.667288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.667398 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.667762 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.667767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.668257 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.668442 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.668454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.669748 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.669753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.671122 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.671151 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.672182 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.672190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.672417 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.673177 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.673202 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.675742 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.679885 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.679891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.679895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.680502 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.681007 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.681013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.681017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.681186 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.681902 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.681907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.682097 (+ 190us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.683377 (+ 1280us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.683705 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663}
I20260620 14:59:33.685139 12731 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:59:33.688751 12582 raft_consensus.cc:348] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
I20260620 14:59:33.689267 12582 raft_consensus.cc:729] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56abbca7b0b497594678803515a8785, State: Initialized, Role: FOLLOWER
I20260620 14:59:33.689759 12582 consensus_queue.cc:260] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.159.253@KRBTEST.COM
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967573, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.252@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
I20260620 14:59:33.695771 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.696471 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:33.697467 12582 sys_catalog.cc:564] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:33.703362 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.683638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54020 (local address 127.21.159.253:40139)
0620 14:59:33.683799 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.683804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.683958 (+ 154us) server_negotiation.cc:407] Connection header received
0620 14:59:33.684087 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.684094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.684169 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.684261 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.686013 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.687178 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.689036 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.690141 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.690294 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.696276 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.696286 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.696292 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.696336 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.700116 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.700953 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.700962 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.700969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.701265 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.701763 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.701769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.701772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.702060 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.702376 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.702705 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.702960 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:59:33.704375 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.685998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.21.159.252:34375)
0620 14:59:33.686212 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.686218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.686390 (+ 172us) server_negotiation.cc:407] Connection header received
0620 14:59:33.686576 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.686580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.686650 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.686746 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.687546 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.688322 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.690013 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.690895 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.691109 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.695957 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.695987 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.695992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.696033 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.699726 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.700872 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.700882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.700888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.701046 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.701482 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.701490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.701496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.702075 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.702261 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.702907 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.703301 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:59:33.705664 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.686066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59924)
0620 14:59:33.686308 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.686322 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.686447 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.686761 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.686765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.687253 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.687408 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.687417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.688440 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.688449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.689823 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.689840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.691148 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.691154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.691352 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.691937 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.691971 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.695716 (+ 3745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.699967 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.699977 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.699984 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.700702 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.701187 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.701193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.701197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.701362 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.702323 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.702330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.702532 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.704690 (+ 2158us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.705150 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260620 14:59:33.710623 22143 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:59:33.711644 12731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: Bootstrap starting.
I20260620 14:59:33.714098 12557 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d56abbca7b0b497594678803515a8785"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042
I20260620 14:59:33.714517 12557 raft_consensus.cc:482] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:59:33.714748 12557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:33.716382 12731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: Neither blocks nor log segments found. Creating new log.
W20260620 14:59:33.716367 12536 tablet.cc:2369] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:59:33.718039 12731 log.cc:824] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:33.718607 12557 raft_consensus.cc:504] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
I20260620 14:59:33.721123 12557 leader_election.cc:288] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [CANDIDATE]: Term 1 election: Requested vote from peers 566e0643eae54a79be278b50b8c22dd5 (127.21.159.253:40139), c9e32d6dbdf84c7897f98e6a289f3d8f (127.21.159.252:34375)
I20260620 14:59:33.717120 12515 master_runner.cc:186] Error getting master registration for 127.21.159.253:40139: OK, instance_id { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" instance_seqno: 1781967573022131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:59:33.735873 12731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: No bootstrap required, opened a new log
I20260620 14:59:33.736176 12515 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:59:33.739629 12751 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:59:33.739923 12751 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:59:33.750742 12731 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
I20260620 14:59:33.751322 12731 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e32d6dbdf84c7897f98e6a289f3d8f, State: Initialized, Role: FOLLOWER
I20260620 14:59:33.751938 12731 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
I20260620 14:59:33.753561 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.754563 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:33.754853 12731 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:33.754993 12706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d56abbca7b0b497594678803515a8785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f"
I20260620 14:59:33.755504 12706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:33.759016 12706 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56abbca7b0b497594678803515a8785 in term 1.
I20260620 14:59:33.759898 12533 leader_election.cc:302] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9e32d6dbdf84c7897f98e6a289f3d8f, d56abbca7b0b497594678803515a8785; no voters:
I20260620 14:59:33.760555 12740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:33.760596 12656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } attempt: 5
Jun 20 14:59:33 dist-test-slave-3mqx krb5kdc[12502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967572, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.159.253@KRBTEST.COM for kudu/127.21.159.252@KRBTEST.COM
I20260620 14:59:33.787781 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.762318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.21.159.252:34375)
0620 14:59:33.762484 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.762490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.762509 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:59:33.762596 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.762601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.762680 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.762787 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.764048 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.765168 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.767031 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.767811 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.767953 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.775280 (+ 7327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.775292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.775300 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.775367 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.780603 (+ 5236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.782753 (+ 2150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.782764 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.782770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.782991 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.783980 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.783988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.783994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.784547 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.784713 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.786760 (+ 2047us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.787171 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:59:33.793535 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.761388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.159.252:34375 (local address 127.0.0.1:59936)
0620 14:59:33.761876 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:33.761902 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:33.762057 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:33.762994 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:33.763001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:33.763614 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:33.763831 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.763845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.765371 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.765377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.766826 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:33.766843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.768216 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.768223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:33.768451 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.769332 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:33.769369 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:33.774794 (+ 5425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:33.781073 (+ 6279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.781083 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.781090 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.782393 (+ 1303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.783333 (+ 940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:33.783342 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:33.783349 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:33.783745 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:33.784925 (+ 1180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:33.784933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:33.785612 (+ 679us) client_negotiation.cc:769] Sending connection context
0620 14:59:33.788718 (+ 3106us) client_negotiation.cc:240] Negotiation successful
0620 14:59:33.790739 (+ 2021us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260620 14:59:33.795993 12740 raft_consensus.cc:686] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [term 1 LEADER]: Becoming Leader. State: Replica: d56abbca7b0b497594678803515a8785, State: Running, Role: LEADER
I20260620 14:59:33.797384 12740 consensus_queue.cc:237] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
I20260620 14:59:33.802915 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5: Bootstrap starting.
I20260620 14:59:33.808352 12656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:33.810173 12656 log.cc:824] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:33.811460 12760 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56abbca7b0b497594678803515a8785. Latest consensus state: current_term: 1 leader_uuid: "d56abbca7b0b497594678803515a8785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.812439 12760 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:33.816773 12770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:33.819025 12770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:33.831557 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5: No bootstrap required, opened a new log
I20260620 14:59:33.839242 12776 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:59:33.850399 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.808099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.21.159.254:46163)
0620 14:59:33.808342 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.808353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.818131 (+ 9778us) server_negotiation.cc:407] Connection header received
0620 14:59:33.819304 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.819310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.819429 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.819510 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.827232 (+ 7722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.828381 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.833953 (+ 5572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.834785 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.834898 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.841764 (+ 6866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.841772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.841776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.841820 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.845816 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.847661 (+ 1845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.847668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.847672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.847840 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.848359 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.848364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.848369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.848764 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.849049 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.849668 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.849950 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 14:59:33.851150 12685 tablet.cc:2369] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:59:33.856156 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.815022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.21.159.252:34375)
0620 14:59:33.815291 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.815301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.822386 (+ 7085us) server_negotiation.cc:407] Connection header received
0620 14:59:33.823451 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.823459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.823554 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.823637 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.830175 (+ 6538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.831430 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.833777 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.834519 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.834633 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.846794 (+ 12161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.846802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.846807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.846853 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.850696 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.853530 (+ 2834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.853536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.853540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.853761 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.854249 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.854255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.854258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.854640 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.854895 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.855497 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.855765 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 14:59:33.860061 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:33.823123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.21.159.253:40139)
0620 14:59:33.823367 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:33.823376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:33.825046 (+ 1670us) server_negotiation.cc:407] Connection header received
0620 14:59:33.826228 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:33.826234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:33.826341 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:33.826428 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:33.831806 (+ 5378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.832979 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.839718 (+ 6739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:33.840407 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:33.840538 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:33.844860 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:33.844869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:33.844874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:33.844930 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:33.852225 (+ 7295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.857241 (+ 5016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.857250 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.857253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.857538 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:33.858070 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:33.858076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:33.858079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:33.858498 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:33.858733 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:33.859358 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 14:59:33.859654 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 14:59:33.847460 12664 master_runner.cc:186] Error getting master registration for 127.21.159.253:40139: OK, instance_id { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" instance_seqno: 1781967573022131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:59:33.880883 12656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
I20260620 14:59:33.881407 12656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566e0643eae54a79be278b50b8c22dd5, State: Initialized, Role: FOLLOWER
I20260620 14:59:33.882010 12656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } }
I20260620 14:59:33.882755 12664 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:59:33.883358 12706 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [term 1 FOLLOWER]: Refusing update from remote peer d56abbca7b0b497594678803515a8785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:59:33.883535 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.884249 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:33.884656 12760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:59:33.886663 12656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:33.886706 12631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d56abbca7b0b497594678803515a8785" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "566e0643eae54a79be278b50b8c22dd5"
I20260620 14:59:33.887089 12630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:33.892660 12630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [term 1 FOLLOWER]: Refusing update from remote peer d56abbca7b0b497594678803515a8785: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:59:33.900098 12631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56abbca7b0b497594678803515a8785 in term 1.
I20260620 14:59:33.902844 12760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [LEADER]: Connected to new peer: Peer: permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:59:33.904687 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56abbca7b0b497594678803515a8785. Latest consensus state: current_term: 1 leader_uuid: "d56abbca7b0b497594678803515a8785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.905009 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:33.918126 12777 mvcc.cc:205] Tried to move back new op lower bound from 7298939182613684224 to 7298939182296428544. Current Snapshot: MvccSnapshot[applied={T|T < 7298939182613684224}]
I20260620 14:59:33.918726 12783 mvcc.cc:205] Tried to move back new op lower bound from 7298939182613684224 to 7298939182296428544. Current Snapshot: MvccSnapshot[applied={T|T < 7298939182613684224}]
I20260620 14:59:33.924109 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d56abbca7b0b497594678803515a8785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.924413 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:33.924518 12760 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d56abbca7b0b497594678803515a8785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.924823 12760 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:33.926751 12770 catalog_manager.cc:1347] Generated new cluster ID: 9441a26e6fac4cffb6d826d64db10591
I20260620 14:59:33.926890 12770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:33.937503 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56abbca7b0b497594678803515a8785. Latest consensus state: current_term: 1 leader_uuid: "d56abbca7b0b497594678803515a8785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.937902 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:33.945230 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d56abbca7b0b497594678803515a8785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.945631 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:33.948328 12786 mvcc.cc:205] Tried to move back new op lower bound from 7298939182613684224 to 7298939182296428544. Current Snapshot: MvccSnapshot[applied={T|T < 7298939182613684224}]
I20260620 14:59:33.953073 12770 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:33.956142 12770 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:33.957461 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d56abbca7b0b497594678803515a8785" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d56abbca7b0b497594678803515a8785" member_type: VOTER last_known_addr { host: "127.21.159.254" port: 46163 } } peers { permanent_uuid: "566e0643eae54a79be278b50b8c22dd5" member_type: VOTER last_known_addr { host: "127.21.159.253" port: 40139 } } peers { permanent_uuid: "c9e32d6dbdf84c7897f98e6a289f3d8f" member_type: VOTER last_known_addr { host: "127.21.159.252" port: 34375 } } }
I20260620 14:59:33.957782 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:59:33.984561 12770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d56abbca7b0b497594678803515a8785: Generated new TSK 0
I20260620 14:59:33.990015 12770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:59:34.014632 12798 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 14:59:34.090221 12738 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.079662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.21.159.254:46163)
0620 14:59:34.079889 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.079895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.087747 (+ 7852us) server_negotiation.cc:407] Connection header received
0620 14:59:34.088083 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.088090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.088218 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.088311 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.089780 (+ 1469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43054: BlockingRecv error: recv got EOF from 127.0.0.1:43054 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260620 14:59:34.094815 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.088033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.21.159.253:40139)
0620 14:59:34.088238 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.088244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.092502 (+ 4258us) server_negotiation.cc:407] Connection header received
0620 14:59:34.093154 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.093162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.093254 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.093332 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.094546 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54028: BlockingRecv error: recv got EOF from 127.0.0.1:54028 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 14:59:34.097793 12735 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.079188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.21.159.252:34375)
0620 14:59:34.079422 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.079431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.091266 (+ 11835us) server_negotiation.cc:407] Connection header received
0620 14:59:34.096264 (+ 4998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.096274 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.096379 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.096513 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.097471 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59948: BlockingRecv error: recv got EOF from 127.0.0.1:59948 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 14:59:34.105329 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.103120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.21.159.253:40139)
0620 14:59:34.103327 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.103333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.103571 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:59:34.103864 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.103872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.104013 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.104142 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.105018 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54042: BlockingRecv error: recv got EOF from 127.0.0.1:54042 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 14:59:34.112139 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.110063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.21.159.253:40139)
0620 14:59:34.110261 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.110268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.110529 (+ 261us) server_negotiation.cc:407] Connection header received
0620 14:59:34.110716 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.110723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.110824 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.110945 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.111898 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54050: BlockingRecv error: recv got EOF from 127.0.0.1:54050 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 14:59:34.121024 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.117878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.21.159.253:40139)
0620 14:59:34.118056 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.118060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.119620 (+ 1560us) server_negotiation.cc:407] Connection header received
0620 14:59:34.119799 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.119806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.119920 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.120042 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.120761 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54062: BlockingRecv error: recv got EOF from 127.0.0.1:54062 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:59:34.130743 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.128424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.21.159.253:40139)
0620 14:59:34.128646 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.128651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.129066 (+ 415us) server_negotiation.cc:407] Connection header received
0620 14:59:34.129337 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.129343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.129480 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.129696 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.130511 (+ 815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54076: BlockingRecv error: recv got EOF from 127.0.0.1:54076 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 14:59:34.144474 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.142332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.21.159.253:40139)
0620 14:59:34.142567 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.142574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.142981 (+ 407us) server_negotiation.cc:407] Connection header received
0620 14:59:34.143193 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.143199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.143293 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.143414 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.144223 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54092: BlockingRecv error: recv got EOF from 127.0.0.1:54092 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 14:59:34.167209 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.165007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.21.159.253:40139)
0620 14:59:34.165200 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.165206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.165685 (+ 479us) server_negotiation.cc:407] Connection header received
0620 14:59:34.165858 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.165865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.165959 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.166094 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.166921 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54102: BlockingRecv error: recv got EOF from 127.0.0.1:54102 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 14:59:34.206362 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.203600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54110 (local address 127.21.159.253:40139)
0620 14:59:34.203810 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.203816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.204525 (+ 709us) server_negotiation.cc:407] Connection header received
0620 14:59:34.204789 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.204799 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.204926 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.205068 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.206074 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54110: BlockingRecv error: recv got EOF from 127.0.0.1:54110 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 14:59:34.277155 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.274826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54124 (local address 127.21.159.253:40139)
0620 14:59:34.275126 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.275134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.275539 (+ 405us) server_negotiation.cc:407] Connection header received
0620 14:59:34.275744 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.275751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.275859 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.276003 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.276922 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54124: BlockingRecv error: recv got EOF from 127.0.0.1:54124 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 14:59:34.414743 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.412718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.21.159.253:40139)
0620 14:59:34.412906 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.412910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.413337 (+ 427us) server_negotiation.cc:407] Connection header received
0620 14:59:34.413642 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.413647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.413729 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.413831 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.414519 (+ 688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54140: BlockingRecv error: recv got EOF from 127.0.0.1:54140 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 14:59:34.678826 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:59:34.677105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.21.159.253:40139)
0620 14:59:34.677323 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:34.677328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:34.677587 (+ 259us) server_negotiation.cc:407] Connection header received
0620 14:59:34.677679 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:34.677684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:34.677766 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:34.677897 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:34.678638 (+ 741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54144: BlockingRecv error: recv got EOF from 127.0.0.1:54144 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:59:34.844097 12776 catalog_manager.cc:1259] Loaded cluster ID: 9441a26e6fac4cffb6d826d64db10591
I20260620 14:59:34.844230 12776 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: loading cluster ID for follower catalog manager: success
I20260620 14:59:34.847082 12776 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: acquiring CA information for follower catalog manager: success
I20260620 14:59:34.849313 12776 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9e32d6dbdf84c7897f98e6a289f3d8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:59:35.020195 12798 catalog_manager.cc:1259] Loaded cluster ID: 9441a26e6fac4cffb6d826d64db10591
I20260620 14:59:35.020331 12798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5: loading cluster ID for follower catalog manager: success
I20260620 14:59:35.024405 12798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5: acquiring CA information for follower catalog manager: success
I20260620 14:59:35.027423 12798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 566e0643eae54a79be278b50b8c22dd5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:59:35.221716 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:35.200057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.21.159.253:40139)
0620 14:59:35.212291 (+ 12234us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:35.212304 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:35.213140 (+ 836us) server_negotiation.cc:407] Connection header received
0620 14:59:35.213355 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:35.213361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:35.213566 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:35.213724 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:35.214300 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.215616 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.217744 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.218498 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.218630 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:35.219506 (+ 876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:35.220277 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:35.220436 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:35.221181 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 14:59:35.221316 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12051,"thread_start_us":12266,"threads_started":1}
I20260620 14:59:35.237835 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:35.229399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.21.159.252:34375)
0620 14:59:35.230824 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:35.230829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:35.230845 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:59:35.230936 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:35.230940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:35.231178 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:35.231317 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:35.231954 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.232842 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.235041 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.235550 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.235674 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:35.236421 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:35.237063 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:35.237175 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:35.237356 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:59:35.237449 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":1168,"threads_started":1}
I20260620 14:59:35.254755 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:35.245676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.21.159.252:34375)
0620 14:59:35.245927 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:35.245937 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:35.246621 (+ 684us) server_negotiation.cc:407] Connection header received
0620 14:59:35.246970 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:35.246978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:35.247130 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:35.247220 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:35.248048 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.249358 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.252158 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.252910 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.253030 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:35.253794 (+ 764us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:35.253961 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:35.254035 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:35.254284 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 14:59:35.254384 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:59:35.257813 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:35.247041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.21.159.253:40139)
0620 14:59:35.247265 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:35.247271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:35.247527 (+ 256us) server_negotiation.cc:407] Connection header received
0620 14:59:35.247918 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:35.247935 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:35.248086 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:35.248172 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:35.249211 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.250970 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.254266 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.255370 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.255515 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:35.256483 (+ 968us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:35.256685 (+ 202us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:35.256795 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:35.257143 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:59:35.257277 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:59:35.265787 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:35.245594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.21.159.254:46163)
0620 14:59:35.258673 (+ 13079us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:35.258683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:35.258710 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:59:35.258838 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:35.258846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:35.259127 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:35.259429 (+ 302us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:59:35.260135 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.261387 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.263003 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:35.263800 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:35.263947 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:35.264516 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:59:35.265024 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:59:35.265158 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:35.265276 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:59:35.265400 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12813,"thread_start_us":12691,"threads_started":1}
I20260620 14:59:35.271688 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 12515
I20260620 14:59:35.272846 12577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:35.387216 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 12515
I20260620 14:59:35.409797 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 12589
I20260620 14:59:35.411130 12651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:35.531261 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 12589
I20260620 14:59:35.552748 22143 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu pid 12664
I20260620 14:59:35.554044 12726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:35.663519 22143 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskmvfD4B/build/asan/bin/kudu with pid 12664
2026-06-20T14:59:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5071 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9764 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375485 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST